Grid-Ireland Operations

DPM 1.7.0 upgrade

2009-06-18T10:08:00.004+01:00

I took advantage of a downtime to upgrade our DPM server. We need the upgrade as we want to move files around using dpm-drain and don't want to lose space token associations. As we don't use YAIM I had to run the upgrade script manually, but it wasn't too difficult. Something like this should work (after putting the password in a suitable file):


./dpm_db_310_to_320 --db-vendor MySQL --db $DPM_HOST  --user dpmmgr --pwd-file /tmp/dpm-password --dpm-db dpm_db

I discovered a few things to watch out for along the way though. Here's my checklist:

Make sure you have enough space on your system disk: I got bitten by this on a test server. The upgrade script needs a good chunk of space (comparable to that already used by the MySQL DB?) to perform the upgrade
There's a mysql setting you probably need to tweak first: add set-variable=innodb_buffer_pool_size=256M to the [mysqld] section in /etc/mysql.conf and restart mysql. Otherwise you get this cryptic error:
Thu Jun 18 09:02:30 2009 : Starting to update the DPNS/DPM database. Please wait... failed to query and/or update the DPM database : DBD::mysql::db do failed: The total number of locks exceeds the lock table size at UpdateDpmDatabase.pm line 19. Issuing rollback() for database handle being DESTROY'd without explicit disconnect().

Also worth noting is that if this happens to you, when you try to re-run the script (or YAIM) you will get this error:
failed to query and/or update the DPM database : DBD::mysql::db do failed: Duplicate column name 'r_uid' at UpdateDpmDatabase.pm line 18. Issuing rollback() for database handle being DESTROY'd without explicit disconnect().

This is because the script has already done this step. You need to edit /opt/lcg/share/DPM/dpm-db-310-to-320/UpdateDpmDatabase.pm and comment out this line:
$dbh_dpm->do ("ALTER TABLE dpm_get_filereq ADD r_uid INTEGER");

You should then be able to run the script to completion.

STEP '09 discoveries

2009-06-08T13:41:00.002+01:00

ATLAS have been giving our site a good thrashing over the past week, which has helped us shake out a number of issues with our setup. Here's some of what we've learned.

Intel 10G cards don't work well with SL4 kernels

We're currently upgrading our networking to 10G and had it mostly in place by the time STEP'09 started. However, we discovered that the stock SL4 kernel (2.6.9) doesn't support the ixgbe 10G driver very well. It was hard to detect because we could get reasonable transmit performance but receive was limited to 30Mbit/s! It's basically an issue with interrupts (MSI-X and multi-queue weren't enabled). I compiled up a 2.6.18 SL5 kernel for SL4 and that works like a charm (once you've installed it using --nodeps).

It's worth tuning RFIO

We had loads of atlas analysis jobs pulling data from the SE and they were managing to saturate the read performance of our disk array. See this NorthGrid post for solutions.

Fair-shares don't work too well if someone stuffs your queues

We'd set up shares for the various different atlas sub-groups but the generic analysis jobs submitted via ganga were getting to use much more time. On digging deeper with Maui's diagnose -p I could see that the length of time they'd been queued was overriding the priority due to fairshare. I was able to fix this by increasing the value of FSWEIGHT in Maui's config file.

You need to spread VOs over disk servers

We had a nice tidy setup where all the ATLAS filesystems were on one DPM disk server. Of course this then got hammered ... we're now trying to spread out the data across multiple servers.

Another day, another globus error

2009-03-05T14:18:00.001Z

After almost 5 years at this lark, I thought I'd got a handle on most of the cryptic globus errors. However, today atlas production jobs started failing with errors like this:

018 (9163559.001.000) 03/05 11:25:27 Globus job submission failed!

  Reason: 22 the job manager failed to create an internal script argument file

Google didn't provide any help, but after asking on LCG rollout, it looked like the problem was the number of files in the relevant user's account. This turned out to be because the script /opt/lcg/sbin/cleanup-grid-accounts.sh that cleans up the grid accounts hadn't run in some days and there were almost 32000 files under that directory.

So there's yet another vital cog in the grid wheel that can fail fairly silently and cause inexplicable errors! Time to add a nagios sensor to check that this cron job runs successfully every night ...

Home-made Torque monitoring

2009-01-08T16:15:00.004Z

I've always been frustrated by the tools for finding out what's going on with Torque/Maui. In particular, it's hard to get an overview of the cluster state. So I compiled up pbs_python and wrote a little web CGI application to provide the information I was interested in. It shows information on jobs running on each cluster node: owner, efficiency, memory usage. It colour-codes the details: grey for under-utilisation and red for over-utilisation. Not perfect but useful for me.

It's available at http://grid.ie/distribution/clustermon

P.S. if something better exists out there, I'd be very interested in hearing about it. I've never found anything that does quite what I want.

Cron Security

2008-12-04T10:08:00.003Z

After the recent Security Challenge we became aware that any pool user could create at and cron jobs on our cluster: obviously not good for security or scheduling.

Initially we wondered if we'd need to create SELinux policies to restrict this but it's much simpler than that. Cron and at support simple allow and deny files to control which users can use the commands. /etc/cron.deny specifies which users are denied access, and /etc/cron.allow specifies which users are allowed. (For full details man crontab.)

In /etc/cron.deny we put:

ALL

and in /etc/cron.allow we put:

   root
   admina
   adminb
   ...

where admina, adminb and so on are the admin users who should have cron access. /etc/at.deny and /etc/at.allow are configured the same way.

This is configured through Quattor. For now we're using the filecopy component to install the config files, but this might be a useful extension to the cron component.

LHC switch-on in Ireland

2008-09-11T08:43:00.002+01:00

We had a great day yesterday at Trinity's Science Gallery where we had a live feed from CERN running all day. There was a lot of press interest and the grid featured heavily due to the fact that the grid group here at TCD makes up half of Ireland's LHC involvement (the other half being the particle physics group at UCD who are in LHCb). We had the GridPP real-time monitor running all day, which provoked a lot of interest and made it onto national TV. One interesting side-effect of all the publicity is that the man on the street now knows that Ireland is one of the few European countries that isn't a member of CERN -- maybe it will cause the politicians to reconsider.

geclipse: a nice grid UI at last?

2008-07-11T10:18:00.003+01:00

I've just been playing around with geclipse and I like what I see. It wraps up the fiddly business of VOMs proxies, information system queries, etc. so you don't have to worry about them. Once I'd downloaded the latest milestone release via eclipse's update manager and set up a VO I was able to submit a job. The WMS was discovered from the information system. They use JSDL to describe jobs, but you fill in the description using dialog boxes -- it can also translate to JDL. There are lots of cool things that I haven't even looked at yet like an interface to amazon ec2 and to local batch systems (to view queues etc.), also visualisation plugins allowing things like interactive jobs.

This looks like a great interface for grid beginners, especially those who're already familiar with eclipse. I knew that sooner or later someone would get round to writing some good software for submitting grid jobs!

Who is that masked user?

2008-04-03T10:21:00.005+01:00

Trying to get a better handle on usage of our cluster, I for the first time realised that Maui actually provides quite a nice way of displaying the efficiency of jobs. It doesn't sort them the way you'd like, but then that's what "sort" is for. Here's the "bottom 10" jobs on our system:


[root@gridgate gridmapdir]# showq -r|sort -n  -k 4|sed -e 's/^[ \t]*//' -e '/^$/d'|head -n 10
JobName  S Par  Effic  XFactor  Q      User    Group    MHost Procs   Remaining            StartTime
447 Jobs     447 of   683 Processors Active (65.45%)
550825_ R DEF   7.53      0.1 DE    fus098   fusion    wn019     1     7:45:00  Fri Mar 28 11:16:42
550438_ R DEF   9.31      0.1 DE    fus098   fusion    wn056     1     1:33:37  Fri Mar 28 05:05:04
550818_ R DEF   9.51      0.0 DE    fus098   fusion    wn072     1     5:19:15  Fri Mar 28 08:50:40
550439_ R DEF   9.65      0.1 DE    fus098   fusion    wn056     1     1:33:37  Fri Mar 28 05:05:04
550429_ R DEF  10.08      0.0 DE    fus098   fusion    wn062     1    00:39:47  Fri Mar 28 04:11:18
550437_ R DEF  10.19      0.1 DE    fus098   fusion    wn056     1     1:33:26  Fri Mar 28 05:05:04
550417  R DEF  10.28      0.1 DE    fus098   fusion    wn011     1    00:27:03  Fri Mar 28 03:58:28
550441_ R DEF  10.30      0.1 DE    fus098   fusion    wn056     1     1:33:40  Fri Mar 28 05:05:04

Looks like I need to find out who this fus098 guy is. Normally my method for doing this is to grep through /var/log/globus-gatekeeper.log but I finally got sick of this and wrote a little python script to translate the funny system used in /etc/grid-security/gridmapdir (documented here) and output the complete set of pool account mappings. I was going to implement all sorts of fancy options for outputting a particular user's mapping etc. but decided I could do what I needed with grep so I'll leave the fancification to someone else. The script is available here and here's some sample usage:

What are the mappings for users with "childs" in their DN?


[childss@gridgate childss]$ ./poolmapping |grep -i childs
dte053:/c=ie/o=grid-ireland/ou=cs.tcd.ie/l=ra-tcd/cn=stephen o. childs:dteam
solovo003:/c=ie/o=grid-ireland/ou=cs.tcd.ie/l=ra-tcd/cn=stephen o. childs:solovo
webcom050:/c=ie/o=grid-ireland/ou=cs.tcd.ie/l=ra-tcd/cn=stephen o. childs:webcom
cosmo007:/c=ie/o=grid-ireland/ou=cs.tcd.ie/l=ra-tcd/cn=stephen o. childs
cosmo004:/c=ie/o=grid-ireland/ou=cs.tcd.ie/l=ra-tcd/cn=stephen o. childs:cosmo
gitest042:/c=ie/o=grid-ireland/ou=cs.tcd.ie/l=ra-tcd/cn=stephen o. childs:gitest

What DN is mapped to dte053?


[childss@gridgate childss]$ ./poolmapping |grep -i dte053
dte053:/c=ie/o=grid-ireland/ou=cs.tcd.ie/l=ra-tcd/cn=stephen o. childs:dteam

But my proxy hasn't expired!

2008-03-11T09:16:00.002Z

We have been plagued with a frustrating problem (especially in our test environment). Users would generate a new proxy, submit a job immediately and then get an error like this:


[childss@ui childss]$ edg-job-status https://cagraidsvr18.cs.tcd.ie:9000/nbPfABOjQHsG7IcFCJcYLg


*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://cagraidsvr18.cs.tcd.ie:9000/nbPfABOjQHsG7IcFCJcYLg
Current Status:     Aborted 
Status Reason:      Job proxy is expired.
Destination:        gridgate02.testgrid.:2119/jobmanager-lcgpbs-test
reached on:         Tue Mar 11 08:47:31 2008
*************************************************************

Which is very annoying as the proxy obviously hasn't expired. It turns out that this is due to old jobs stuck on the RB (whose proxies have expired). The problem can be cleared by logging onto the RB, identifying old jobs for the user's DN and removing them using condor_rm. I'll leave it to someone else to explain why this arises. I hope it's been fixed in the new WMS.

Can Quattor save the world?

2008-02-12T08:05:00.000Z

Due to the wonders of planet, I've just seen this post by Andrew from Glasgow with the intriguing comment: "Are there any better tools? (is Quattor the savoiur for this type of problem)". This post was due to the frustrations of cobbling together fabric management from a collection of very good, but separate tools. So I thought I'd briefly describe some of the advantages of Quattor. I know many were burned in the early days of Quattor by its complexity and obscurity, but times really have changed and I suggest you revisit it. So here are just a few of the reasons I like it:

It's got a real programming language: this gives you data structures (e.g. hashes), types (allowing validation of the values you type in -- i.e. it will recognise that "123.34.32.O7" (spot the deliberate mistake) isn't an IP address).
The gLite configuration is up to date with YAIM (and often ahead of it): Michel Jouvin has led the way on a number of deployment issues in LCG (e.g. 64-bit WNs, space tokens, etc.) and all this stuff gets into Quattor before YAIM. (Also DNS-style VO-names, Xen configuration, etc., etc.) We have found that whenever we have to do something non-custom (e.g. publishing multiple different jobmanagers from one CE in GIP) it's a doddle in Quattor due to the availability of proper data structures (see above).
The Quattor Working Group templates are effectively a complete Grid distribution in a way that gLite itself isn't. What I mean is that they provide all you need for going from bare metal to installation of a complete SL-based Grid site. This is ideal for new/small sites.
It's a true community effort: having been involved in YAIM development for MPI, I have first-hand knowledge of the protracted process involved in getting anything fixed in gLite. In contrast, Quattor functions as a true OSS project: if there's a problem, you fix it and check it in. If it passes muster after a lightweight review, it's included in the core release. Problem solved.
It provides integration with installation and monitoring: the contents of configuration profiles for a machine are directly used to generate Kickstart templates, and monitoring (using Lemon is also tightly integrated with a raft of sensors and alerts available.

Play it again, SAM

2008-02-04T10:04:00.000Z

After much pain, we have finally got a SAM server up and running for Grid-Ireland (see here). We used to run an SFT server, but it was ancient and when eventually the client software became incompatible with the UI distribution, we decided to move to SAM. It looked like there were quite good installation docs available so we assigned it to someone as a Friday afternoon project. That was two months ago! It turned out that the documentation, while good, had a few critical errors/omissions in it, and the support was non-existent. We've finally got it sorted now (the last problem was solved when I divined by reading the source code that you had to define an ACL of approved DNs in the config file) and it looks like it should be useful in keeping track of our non-EGEE sites. We'll try and feed our experience back upstream, or (probably more useful) stick it on a public page so it makes it into Google.

Stepping through the pgrade portal

2008-02-01T09:11:00.000Z

As a Grid veteran, I normally submit jobs using edg-job-*, and at this stage I've almost given up hope that there could be a less painful way of getting jobs onto the Grid. I've tried Ganga in the past, and it was promising, but it didn't work well with the broken MPI on the EGEE grid, so I kind of gave up on it. The latest thing we've installed is the p-grade portal which has been around for a good while and is allegedly getting "mature" now. The first problem after creating an account was getting my cert set up for use in the portal. I had the cert and key on my local machine, and tried to upload them to a MyProxy server to get something the portal could use. At this point, I was asked for the hostname and port number of the MyProxy service. Now, I actually administer the MyProxy server, and still had to ask a colleague which port it ran on. There is no way in the world a user should have to know this, but apparently you can't set defaults in the portal. We're running version 2.5 still so maybe it's fixed in 2.6.

Once I got my cert up and running, I went to submit a job. My first job, the challenging "/bin/hostname" test failed. I didn't expect that. Apparently pgrade uploads a binary from your local machine by default rather than executing something hosted on the remote machine. As my local machine is FC6 and the execute node is SL3, the uploaded hostname binary wouldn't run. So if you wanted to run the hostname program on the remote host, you would have to upload a script which ran /bin/hostname.

The next challenge was how to add input files to the job. It turns out that this is done by adding "ports" to the job node you define. Everything in pgrade is a workflow, so files are ports that allow data to flow between nodes (or from the local machine). It takes a little while to get used to this approach.