Victoria Hermes II Tests

Introduction

Purpose was to run and also benchmark jobs on Hermes II. Input datasets used were triumf19Aug2010.txt which consisted of the following:

mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
total files: 1971
total size: 2.01 TB
total events: 9851996
> dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210/
dq2-list-datasets-container mc09_7TeV.105830.JF17_herwig_jet_filter.merge.AOD.e507_s765_s767_r1215_r1210_tid125847_00
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
total files: 1000
total size: 755.12 GB
total events: 4994464
> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1215_r1210_tid125845_00
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
total files: 500
total size: 757.96 GB
total events: 4994464
> dq2-list-datasets-container mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210/
mc09_7TeV.108087.PythiaPhotonJet_Unbinned17.merge.AOD.e505_s765_s767_r1207_r1210_tid124453_00
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
total files: 1000
total size: 777.51 GB
total events: 9996249
> dq2-list-datasets-container mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210/
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123082_00
mc09_7TeV.105001.pythia_minbias.merge.AOD.e517_s764_s767_r1204_r1210_tid123186_00

Note there are 5 datasets in those 4 containers. (Note: the above datasets reside in LOCALGROUPDISK and so new HC sites reflecting this need to be created.) Additional HC configurations: uncheck "Resubmit enabled" and set "Num datasets per bulk = 5". For Panda, filesize set by Dan/Johannes to -o[defaults_DQ2JobSplitter]filesize=13336 (in MB). For Ganga, configure HC to submit with filesize=13336 so that the number of jobs will match that of the Panda test.

Hermes II cluster

The Hermes II nodes have IB, 12 cores and 24 GB of RAM. The current memory settings for the atlas queue are such that only 11 jobs fit on each 12-core node (since some memory on the node is used by the OS, and the formal ATLAS requirement is 2 GB per job.) With HC we can compare running 11 or 12 jobs per node and make sure that memory usage isn't constrained too much.

See CACloudSiteHardware for details on the Hermes I and II nodes.

Tests

Test 9457

Test 9457 Using 8 Hermes II worker nodes in the atlas-test queue, so 88 HC jobs will run at a time since the memory constraints place only 11 jobs on each node. Started running around 13:27 Aug. 16 2012. At the time, there were around 360 prod jobs running and 340 regular analysis jobs. 612 of those jobs were running on IB nodes. The local disks in the Hermes II nodes have much better performance than those in the Hermes I nodes, but they are still a bottleneck (writing at 100 MB/s) when jobs are starting. Hopefully this will be ameliorated when we have CVMFS. The active memory usage is usually less than 10 GB per node, but cache fills the RAM all the way up to 24 GB. CPU usage is about 85% - 90% at peak, with 11/12 cores used. The pool nodes have significant spikes of CPU usage, but that started before this HC run so it must be some other job activity. Unfortunately, we do not have any IB traffic monitoring in place yet.

Test 9467

Test 9467 Using 6 Hermes II worker nodes in the atlas-test queue, so 72 HC jobs will run at a time. This time, the default memory requirements are slightly less, so 12 jobs will fit on a node. The pool nodes are significantly less busy (both CPU and eth0 bandwidth, but we don't know about ib0 bandwidth) during this HC test. It may be because of the background environment of ATLAS jobs, or perhaps the DDN controllers are reaching their limits, since the entire cluster is quite busy now. In any case, this seems to be more of an effect than the increased memory or disk contention from having 12 jobs per node instead of 11 (which didn't look too significant in the node stats).

Test 9513

Test 9513 This will run on 4 "Hermes Classic" nodes. The performance is slightly lower than an earlier HC on the Hermes I nodes. This may be because the cluster (esp. dCache and the NFS server) are busier now with a much higher number of jobs running. But the 'download input files' time is faster now than before; not sure why.

Results

Performance-wise, there is no reason not to decrease the default memory requirement of ATLAS jobs so that 12 jobs fit on each of the new nodes.

Compared to an earlier HC on the Hermes I nodes, the performance of the new nodes is roughly ~ 5-10% better in most metrics. This is approximately commensurate with the performance difference of the CPUs, as measured by HEP-SPEC06 at least.

Compared to the recent HC on Hermes I nodes, the performance of the new nodes is roughly ~ 20% better in most metrics. This may be a more fair comparison, since both of these tests were taken with ~1000 background jobs running, whereas the Feb 2011 test would have had only ~300 background jobs running.

Overall, the IB-enabled nodes have a faster 'download input file' time. But the 'software setup time' is significantly slower - could that be because the IB-enabled nodes reach the NFS server by going through the default gateway, and then jumping on the ethernet network?

-- AsokaDeSilva - 2012-08-16

Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r11 - 2012-08-20 - RyanTaylor
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback