Testing ALRB and manageTier3SW before deployment

This information is not for users site admins; it is for ATLAS software experts for administration.

This section tells you how to make changes in the ATLASLocalRootBase (ALRB) and the manageTier3SW packages, how to test before deploying the changes.

Stop Please be very careful as all of ATLAS (Tier1/2 grid sites, Tier3 sites, cernVM and lxplus) automatically update based on the latest tagged versions and depend on reliability of this software.

Note About Supported Platforms

Installed on SL6 x86_64 SL7 x86_64 Comments
SL6 x86_64 X    
SL7 x86_64   X see known issues

Note About Software in GitLab

The software packages are in the gitlab repository at CERN and can be seen on this link. They are:

  • ATLASLocalRootBase: the software UI which the user interacts to do various tasks;
  • manageTier3SW: the software the Tier3 manager will use to install software, including ATLASLocalRootBase.
  • Tier3SWConfig: the configurations for what to install at sites;
You need to have write access to the git repository to checkout/checkin changes.

Step 1: Testbeds

TRIUMF Testbeds

There are VMs (hosted on atlasuser1 which depend on the Tier-1 infrastructure) available as testbeds; the VM names are self-explanatory. (Access to these machines are under TRIUMF Tier-1 policy which does not allow non-TRIUMF personnel access):

  • atlas-sl6x64
  • atlas-sl7x64
On each of these machines,
  • login as atlasadmin to run updateManageTier3SW with the appropriate test tags.
  • login with your own private account to test by first setting
  export ATLAS_LOCAL_ROOT_BASE=/ALRB/atlasadmin/`hostname -f`/ATLASLocalRootBase
  alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh'
  • the atlasadmin account is to be used only for running updateManageTier3SW. Nothing else should be done there.

HELP These VMs are also used to test new versions of cvmfs so they are routinely destroyed and recreated; the user areas are preserved though so this is transparent. (ie please let the other users know if you are using a testbed so that your usage is not disturbed.)

CERN Testbed

You also need to test on lxplus in addition to the above vanilla Scientific Linux OS machines

There is a VM that is accessible for installations by authorized users.

  • login to lxplus as yourself and then do
  • ssh aibuild003
  • sudo -i -u atlast3
  • You can run updateManageTier3SW.sh which will update the software on /afs/cern.ch/work/a/atlast3/public/ATLASLocalRootBase.
  • The atlast3 account is meant only for installation purposes. Nothing else should be done as atlast3.

To test your changes, first put this inside your .bashrc and .zshrc files of your own lxplus accounts.

export RUCIO_ACCOUNT=<your lxplus name>
alias testALRB='source /afs/cern.ch/work/a/atlast3/public/scripts/alrbTest.sh'
  • login to lxplus as yourself
  • type testALRB so that you switch to using the ALRB installation done by atlast3 as described above. You can now do setupATLAS etc.

Other Testbeds

If you prefer to use your own machines as testbeds, this is how to do it - for example on SL7:

Creating Your Own Instead of Using the VMs

  • It is best to create a separate atlasadmin account for Tier3 software installations so that permission issues of any software can be flushed out when accessed as a regular user (ie from your own account).

  • Create a file to override the ALRB definitions; eg. testALRB.sh containing:
    alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh'

  • Checkout manageTier3SW and install the software
git clone https://gitlab.cern.ch/atlas-tier3sw/manageTier3SW.git ~/userSupport/manageTier3SW
cd ~/userSupport/manageTier3SW
./updateManageTier3SW.sh -a $HOME/testALRB

  • Updating: you will be able to simply type source updateManageTier3SW.sh to update to the production version.

  • From this point onwards, you will be able to simply type source ~/testALRB.sh which will then use the testbed version to validate the software (and use the existing Athena kits as well in your tests.)

Step 2: Making Changes

Initial Checkout of Repos

You will need to have git checkin privileges to do this for the ATLAS Tier3 SW repo. Make sure that your git account at gitlab.cern.ch has your ssh keys registered.

On your testbed (with own account), create a directory for code development.

mkdir -p $HOME/Tier3Dev

Next, checkout the code that you need to change (most of you will need only the Tier3SWConfig repo so you can clone only that);

cd $HOME/Tier3Dev
git clone ssh://git@gitlab.cern.ch:7999/atlas-tier3sw/manageTier3SW.git
git clone ssh://git@gitlab.cern.ch:7999/atlas-tier3sw/ATLASLocalRootBase.git
git clone ssh://git@gitlab.cern.ch:7999/atlas-tier3sw/Tier3SWConfig.git

Changing Code in a Repo

In this example, I show how to change code (note you will be working with the master branch):

cd $HOME/Tier3Dev/<repo>
git pull
# (make changes)
# make an entry in RELEASE.NOTES
git status # check what is changed
git commit -am "some comment"
git status # check what is not committed
git push
git tag <tag> 
git push origin <tag>

In the above, example, <tag> should be a "candidate" tag unless the repo is ATLASLocalRootBase in which we provide a production tag. Candidate tags are of the form cand-<tag>-v<integer> where the <tag> is the release which will go into production and <integer> is a incrementing number.

TIP You can see the tags already used by git tag when in the repo dir.

Case 1: Updating Versions of Tools (eg Panda, CMake, etc)

Most of the time, changes will involve bumping up versions of 3rd party software such as Panda, Ganga etc. This involves making changes in the appropriate file(s) in the package directory Tier3SWConfig /tools.

Note that the tools are usually from a repository somewhere - either from the developers or saved by us at CERN (http://atlas-tier3-sw.web.cern.ch/atlas-Tier3-SW/repo/). The repository at CERN is accessible to authorized people and accessible through /eos/project/a/atlasweb/www/atlas-Tier3-SW. Note that there is a file /eos/project/a/atlasweb/www/atlas-Tier3-SW/DevelopersLocations.txt detailing where to copy new versions from in case the developers omit this information.

HELPYou can test and deploy new versions of 3rd party software without making it the default or testing. This way, the new versions can be published on cvmfs and be tested further by others as well without affecting production. setupATLAS; showVersions tells you how to setup the non-default versions.

Case 2: Updating ATLASLocalRootBase Code

If you do bug fixes or other changes to ATLASLocalRootBase, you can first commit those changes, give the package a production tag and then make the version change in the file Tier3SWConfig /tools/alrb.sh.

Case 3: Updating manageTier3SW Code

(Note, you will rarely have to modify this.)

If you do bug fixes or changes to manageTier3SW, you can commit those changes (do not change the latestVersion file until you finish testing); give it a private tag and use that private tag to test as directed below.

Step 3: Testing

This should be done on all testing platforms including lxplus.

This is done preferably as 2 different accounts, one for installation and one for testing:

Update the Software on Testbed

Login as

  • atlasadmin on the TRIUMF VMs
  • atlast3@aibuild for the CERN VM

Now, depending on where the changes were made and what you want to test, updateManageTier3SW [--help] has several options that can be used.

Assuming you have the most common case, where you updated a tool in Tier3SWConfig, you will do something like:

updateManageTier3SW.sh -c <candidate tag>

which will pull in the changes of Tier3SWConfig tagged as <candidate tag> and test it with the other software unchanged.

User Testing

Now in a fresh terminal session,

  • login to your own account on the VMs at TRIUMF to get the testing ATLASLocalRootBase environment
  • login to lxplus and type testALRB to get the testing ATLASLocalRootBase environment
Then test the newly installed software or tools. This is an example on how to use toolTest. For example, to test PandaClient:
setupATLAS  # in your account but accessing the ATLASLocalRootbase installation that you own for testing
toolTest -m test panda

For example, toolTest -h panda will show you what environment variables can be exported prior to running the tool to change the versions tested. v is a verbose option that will show you details of the tests as they are done.

toolTest -m option Shells Tool Version Tested Comments
user same as user's shell Production For users
test all shells Testing For checking testing versions
validate all shells Production For validating production versions or Containers
toolTest an be used to test everything at once (eg for any new containers that are generated). eg toolTest -m validate all.

Step 4: Publishing

When you are satisfied with the tests, go back to your development area directory, update latestVersion file and give it a production tag that matches the latestVersion file contents.

cd $HOME/Tier3Dev/<repo>
git diff @{upstream}  # make sure that there are no changes that were not tested since we are dealing with master branches
# edit and put an entry in RELEASE.NOTES if not done previously
# edit latestVersion and ensure that that will have a production tag as described in the RELEASE.NOTES entry
git status # check what has changed
git commit -am "latest version"
git status # check that you have committed everything
git push 
git tag <tag>
git push origin <tag>

Step 5: Final Check and Deployment

As a last sanity check, login to your testbed in a fresh terminal session and run updateManageTier3SW.sh It should pull in the latest tag of manageTier3SW and try to update packages.

The cvmfs server automatically runs manageTier3SW every 6 hours so software gets deployed on its own. However, if you want it faster, please login to the cvmfs server and update/publish the changes.

(from lxplus)
ssh cvmfs-atlas
sudo -i -u cvatlas
cat .install.lock. date; ls -la .install.lock  # check for lock files and wait until there are none.
# optional, if you need to let others know:  copy the last 2 lines of tier3update output that give you the timestamp of when the changes will be available.

Step 6: Notify

Mail to "atlas-adc-tier3sw-install (ATLAS Tier3 software installation requests and discussions)" <atlas-adc-tier3sw-install@cern.ch> a note saying what the changes are an paste the last 2 lines of the tier3update output showing when the changes would be on cvmfs.

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2019-06-09 - IsabelTrigger
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback