Containers

Introduction

Containers allows one, for example, to setup a SLC6 or centos7 OS on a host running a different OS with very little overhead. setupATLAS encapsulates the container startup in Linux or MacOSX so that even if, for example, you are running Ubuntu or a Mac you will run in an SL6 or SL7 ATLAS ready OS environment.

These are the supported containers and OS platforms:

Container Software Linux MacOSX
Singularity DONE Suported No Unsupported
Docker No Not tested DONE Supported
For Singularity on MacOSX, since one has to first install a VM (Singularityware Vagrant Box), I do not see much point in this since users can just as well install CernVM. So Docker is preferred in this situation.

Important Notes

Note There are some things users need to be aware of when using containers.

Be realistic with your laptop/desktop expectations. In principle containers should allow you to most efficiently utilize your resources but these also depend on how they are used by your host machine (eg running Mail, web browsers etc); machines where the CPU is dedicated to running jobs see only tiny degradation (< 5%) compared to running on bare metal. It will use cvmfs so if you have a nearby squid, configure that in your host machine's /etc/cvmfs/default.local to speed things along.

  • All platforms:
    • Note that your home directory will be mapped to /home/<username>.
    • If you use symbolic links, use relative ones instead of absolute ones (if possible) since the paths in the container can be different for $HOME.

  • MacOSX:
    • File names are case insensitive on OSX (default format file system) which is not the case on Linux. If your host is a Mac, then use unique case-insensitive file and dir names.

Installation

Container Software

First, decide on the container software to use. There are 2 choices:

  • Singularity (default except for MacOSX)
    • Enable namespaces
      echo "user.max_user_namespaces = 15000"  > /etc/sysctl.d/90-max_user_namespaces.conf
      sysctl -p /etc/sysctl.d/90-max_user_namespaces.conf
    • Singularity is now distributed through cvmfs so you no longer need to install it yourself.

  • Docker (default for MacOSX)
    • Install Docker https://www.docker.com/community-edition
    • Please update to Docker 18.06.1 or newer as this has been tested.
    • (MacOSX) In Docker Preferences, under File Sharing, add these /Users /Volumes /cvmfs /private /tmp

cvmfs

  • Install cvmfs.
    • You should not let automount unmount the cvmfs repos.
      • eg for Linux, /etc/auto.master should have a no timeout: /cvmfs /etc/auto.cvmfs --timeout 0
    • Relocated cvmfs will also work.

Miscellaneous

  • All Platforms
    • Create $HOME/.globus and copy your grid credentials. Make sure that the protections are correct
      • chmod 744 $HOME/.globus
      • chmod 444 $HOME/.globus/usercert.pem
      • chmod 400 $HOME/.globus/userkey.pem
    • Define RUCIO_ACCOUNT either in your $HOME login script or in the $HOME/.*rc.container scripts (see Login Scripts section below)

  • MacOSX:
    • Install XQuartz https://www.xquartz.org
      • Launch XQuartz preference, select security and enable "Authenticate connections" and "Allow connects from network clients"
    • In System Preferences, select Sharing, entry a name in "Computer Name". Give it a name without spaces or special characters - only A-Za-z0-9 in the name.

non-cvmfs ATLASLocalRootBase

ALERT! This is not recommended unless you know what you are doing (eg for HPC resources).

ALERT! If you specify the +nocvmfs option, eg setupATLAS -c <container>+nocvmfs, cvmfs will not be available from /cvmfs but ATLASLocalRootBase will now be available from /alrb/ATLASLocalRootBase. This version will be the same as that on the host machine - if setupATLAS was from cvmfs, then that from cvmfs will be mounted on /alrbATLASLocalRootBase. Alternately, if setupATLAS was from a local ATLASLocalRootBase installation, then the container's /alrb/ATLASLocalRootBase will be from the host machine's local version.

If you want to use a local ATLASLocalRootBase installation in the container instead of that from cvmfs,

  • you will need to specify the +nocvmfs option (eg setupATLAS -c <container>+nocvmfs) which will avoid mounting any cvmfs repos in your container,
  • use setupATLAS -c <container>+nocvmfs from your local ATLASLocalRootBase installation.

  • This means that you will be responsible
    • for the installation and regular updates (at least once/day) of the host machine's local ATLASLocalRootBase installation,
    • ensuring that the tools for the same architecture of the containers you use are installed on the host machine's local ATLASLocalRootBase installation.

  • Since grid-middleware CA/CRL updates are only possible on Linux architectures, a MacOSX host machine's local ATLASLocalRootBase will not be fully functional for grid related tools. There is no problem for Linux based host machines.

This table summarizes what is and is not possible for Linux containers:

Host Machine OS Container ALRB is Host cvmfs Container ALRB is Host cvmfs Container ALRB is Host Local Install
  Container /cvmfs is available +nocvmfs option used
Container /cvmfs NOT available
+nocvmfs option used
Container /cvmfs NOT available
Linux DONE DONE DONE
Mac DONE DONE No (grid tools will not work)

Setup

To setup from a host machine that has the software installed, simply do eg setupATLAS -c sl6 (if your machine is part of a cluster and has batch queue access and you want to submit batch jobs, do instead eg setupATLAS -c sl6+batch)

setupATLAS -c <container> where container can be slc5 (unsupported), sl6, sl7, slc6, centos7, rhel6 or rhel7 or any string that contains these case insensitive keywords to setup the default container.

Alternatively, <container> can be a path to a container file (Singularity) or from the registry (Docker). An example of setting up a docker container with URL that will run in Singularity: setupATLAS -c docker://atlas/analysistop

Note that Docker will download and the container files from the registry the first time that container is run or if it needs refreshing; Singularity will use the files distributed on cvmfs unless another location is specified.

Singularity Default Containers

ALERT! Singularity will use the unpacked containers in /cvmfs/atlas.cern.ch/repo/containers/fs/singularity by default.

However, if you run Singularity versions 2.4.1 and newer, and if you prefer to use the images (cvmfs/atlas.cern.ch/repo/containers/images/singularity) then you can select that with a keyword images separated by a plus; eg setupATLAS -c images+slc6

Login scripts

The container will not use the login scripts in your $HOME dir. If you want to customize your environment, then create these files and they will be used for the container automatically:

  • $HOME/.bashrc.container (for bash) or $HOME/.zshrc.container (for zsh)
# example of .bashrc.container to set the prompt to pwd
export PS1="`pwd` $ "

# example of .zshrc.container to set the prompt to pwd
export PROMPT="`pwd` $ "

  • $HOME/.ssh.container if you need a different configuration (eg for MacOSX hosts since Linux is different); otherwise $HOME/.ssh will be used.
  • $HOME/.globus will be used
  • some environment variables will be passed from your host machine to the container; among the important ones, if they exist, are $RUCIO_ACCOUNT, $SITE_NAME and settings for X11 forwarding.

ContainerTools

After the container starts, you will have access to these tools (-h for usage):

Command Description
batchScript Generate a script for submission to the batch cluster which should have the container software (singularity/docker) available for users. Note in particular the -o and -O options.
lsh For a dir or path on the container, show the corresponding on the host machine. This is useful if you need to use eg emacs on the host machine to edit a file that you know the path inside a container but when the bind mounts are different from the host machine's mount points.
eval "echo <path on host> | sed $ALRB_CONT_SED2CONT"
will convert a path on the host machine to one on the container

Advanced Options

These environment variables affect the behaviour of the -c/--container option (from setupATLAS -h):

Environment Variable Description
ALRB_CONT_OPTS Additional options to pass to singularity or docker; eg. singularity [opts]
ALRB_CONT_CMDOPTS Additional options to pass to singularity or docker commands; eg. singularity exec [opts]
ALRB_CONT_SWTYPE Container software (Singularity/Docker) to use. default: Docker (MacOSX), Singularity (others)
ALRB_CONT_PRESETUP Pre-setupATLAS commands to run in container. Multiple commands can be semi-colon separated or a file to execute or source.
ALRB_CONT_POSTSETUP Post-setupATLAS commands to run in container. Multiple commands can be semi-colon separated or a file to execute or source.
ALRB_CONT_RUNPAYLOAD Run these commands and exit the container. Multiple commands can be semi-colon separated or a file to execute or source. Use this mechanism in a script to run as a batch job.
ALRB_CONT_CONDUCT Comma delimited keywords affecting bahaviour; Recognized values:
* dockerJoin: join an already running docker container; see FAQ
ALRB_CONT_USERCONFIGBATCH The location of a text file containing configurations you want to add into batchScript (see previous section for batchScript)

Setting Advanced Options In Site Config File

If $ALRB_localConfigDir is defined, it will look for a file alrb_container.cfg.sh and use those configurations to set up a container. To make it simpler to change the above environment variables without having to check for their existence, or to consider too many back-slashes, you can use the syntax set_ALRB_ContainerEnv inside the alrb_container.cfg.sh to set the variables. For example, see this.

In addition, if run_ALRB_UserContainerConfig is present in the site alrb_container.cfg.sh file, the user's configurations will be setup as well. This will happen only once.

Setting Advanced Options for Users

Users can define a file $HOME/alrb_container.cfg.sh setting up the env variables in the table above to configure the container. The syntax set_ALRB_ContainerEnv can also be used inside this file. This will be run only once, either by the site config file or else when the container sets up.

# example of $HOME/alrb_container.cfg.sh on lxplus (note afs and eos will get mounted by default in the near future on lxplus;  
#  this is for the interim).
# make all of afs available
if [ -d /afs ]; then
    set_ALRB_ContainerEnv ALRB_CONT_CMDOPTS "-B /afs:/afs"
fi

# make all of eos available
if [ -d /eos ]; then
    set_ALRB_ContainerEnv ALRB_CONT_CMDOPTS "-B /eos:/eos"
fi

# set HOME to be the same as that of lxplus (ie afs since it is mounted now)
set_ALRB_ContainerEnv ALRB_CONT_POSTSETUP "export HOME='$HOME'"

Batch Jobs

If your cluster allows for batch jobs (eg slurm, condor or lsf) to start singularity or docker containers, you can submit and access the batch queues from inside your container interactive session. For example, if you type

setupATLAS -c slc6+batch

You can next generate a script to submit to the batch system. eg.

batchScript "source <somepath in container>/myJob.sh" -o submitMyJob.sh

(The above will create a file submitMyJob.sh that will run myJob.sh inside the same type of container you are on now but on the batch queue.)

(You can also simply do this: batchScript "\echo \"Starting myJob\"; cd myDirSomewhere; asetup 19.0.3,slc6,here; athena ...." -o submitMyJob.sh)

Finally submit it from inside the container with one of these:

  • bsub -L /bin/bash submitMyJob.sh if your site supports lsf
  • sbatch --export=NONE submitMyJob.sh if your site supports slurm
  • condor_submit <jdl file executing submitMyJob.sh> if your site supports condor

Modifier Summary

This table summarizes the modifiers you can add to setupATLAS -c <container> using the + separator:

Modifier Description
+batch If the host machine has batch queue commands, these will be made available inside the container.
+images Instead of a unpacked filesystem from cvmfs, mount the container image from cvmfs. Note that this will reduce efficiency.
+nocvmfs /cvmfs will not be mounted inside the container. ATLASLocalRootBase will be available instead from /alrb/ATLASLocalRootBase. This will be the installation from the host machine's /cvmfs or, if that is unavailable, from the host machine's local ATLASLocalRootBase installation. Note: do not use this option unless you know what you are doing and why you need it.

FAQ

Mount point summary

On Host (if defined) On Container
$TMPDIR /scratch
$PWD /srv
$HOME /home/<user>
/cvmfs /cvmfs
- /alrb (temporary startup dir)

MacOSX: SSH keys in Docker container

There is a known issue with Docker containers and MacOSX that one cannot use the ssh-agent from MacOSX inside the Linux container. If you want to use ssh keys, then do ssh-add (or ssh-add $HOME/.ssh/<keyfile> if the keyfile is not named the default id_dsa or id_rsa) after the container starts to add your key to the container's ssh-agent. Passwordless logins should work again after that.

All platforms: X11 and ssh

If you ssh to a host that has singularity, you can start it up and portback X11 as follows; eg.

  • ssh -AY <hostname>
  • setupATLAS -c slc6
  • MacOSX local machine, start XQuartz and wait until it starts.
  • on your local machine, xhost + localhost

All platforms: Missing favourite software

These containers are lightweight and may be missing software that Tier3 users may take for granted. But there are easy ways around this.

For example: emacs: Keep in mind that your file system is shared with the container and so you can use emacs on your host machine. Similarly for web browsers.

If there is something you think should be made available, and cannot be used from the host machine, do let us know through the DAST eGroup.

All platforms: Why is my dir /srv after startup

If you do setupATLAS -c <container> in a directory, that current directory gets mapped to /srv on the container. (This is intentionally done). So If you are starting from somewhere other than your $HOME directory on the host, you will end up on /srv when the container starts for consistency. Your $HOME dir in the container is still there and unchanged (you can type cd to go to it).

Docker: Start an Interactive Session in a Running Container

You can choose to automatically start a new container or join an already running container (the latest if you have multiple versions of the same one running). This will save resources and is done automatically if an environment variable is defined to affect the behaviour as shown below. (This is not made default behaviour as users need to be actively aware of doing it because of the caveat below.)

export ALRB_CONT_CONDUCT="dockerJoin"   # join an already running docker container or start a new one otherwise 
setupATLAS -c <container>

ALERT! Caveat: If you are joining a running container you will see this warning that the dependent session will die once the main container is exited.

$ setupATLAS -c slc6
Docker: Join already running my-atlas-grid-centos6-1519434423
        Remember that if you exit that container, this session will die

Tier3 Sites And Defining setupATLAS

This section is for Tier3 admins who want to define their sites to use containers and setupATLAS

Sites can source a script such as the following. Note the export ALRB_containerSiteOnly=YES which will prevent setupATLAS tools from being available except if it is inside a container (this is to check those users forgetting to start a container). The environment ALRB_containerSiteOnly is for those sites whose host OS is not ATLAS ready and want all users to run ATLAS jobs or tools inside the container.

These are the recommended steps:

  • create a common directory somewhere that all users can access and which only ATLAS administrators at your site can edit. In the example below, I call this directory /atlasadmin.

  • create a setup script like the one below that all your uses will source (in ~/.bashrc or ~/.zshrc); for example, I call this /atlasadmin/AtlasUserSiteSetup.sh. Your ATLAS users should add this line in their login scripts source /atlasadmin/AtlasUserSiteSetup.sh.

if [ ! `typeset  -f setupATLAS > /dev/null` ]; then
    unalias setupATLAS  > /dev/null 2>&1
    # eg at Compute Canada sites, $CC_CLUSTER is defined and we use that for local site configurations
    my_AtlasSiteDir="`dirname ${BASH_SOURCE:-$0}`/$CC_CLUSTER"
    function setupATLAS
    {    
        if [ -d  /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase ]; then

            export ALRB_localConfigDir=$my_AtlasSiteDir
            
            # uncomment if all ATLAS users must use containers interactive (eg not ATLAS-ready site)
            #export ALRB_containerSiteOnly=YES

            # modify if you have a site name or are co-located with a Tier2
            #export SITE_NAME=

            # modify if, for example, singularity lives elsewhere
            #export PATH="/opt/software/singularity/bin:$PATH"

            # modify if additional local Tier3 directories need to be bind mounted inside the container   
            #\echo $ALRB_CONT_CMDOPTS | \grep -e "/project:" > /dev/null 2>&1
            #if [ $? -ne 0 ]; then
            #    export ALRB_CONT_CMDOPTS="$ALRB_CONT_CMDOPTS -B /project:/project"
            #fi
 
            export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
           
            source $ATLAS_LOCAL_ROOT_BASE/user/atlasLocalSetup.sh $@
            return $?
        else
            \echo "Error: cvmfs atlas repo is unavailable"
            return 64
        fi
    }
fi

  • (Optional) If the cluster allows batch jobs, add a file /atlasadmin/siteConfigBatch.txt containing the additional commands needed by the site for the batch job. The following is an example for Compute Canada Tier3s. This file will be included in the executable of the batch job generated by batchScript.
#----------------------------------------------------------
# Compute Canada site setup for ATLAS
#----------------------------------------------------------
module load singularity
source /project/atlas/Tier3/AtlasUserSiteSetup.sh

# Reference: https://docs.computecanada.ca/wiki/Running_jobs#Accounts_and_projects

# If you have _more_ than one allocation, configure your ~/.bashrc on the
#  login machine to have these lines (and login again)
#      export SLURM_ACCOUNT=<def-someuser>
#      export SBATCH_ACCOUNT=$SLURM_ACCOUNT
#      export SALLOC_ACCOUNT=$SLURM_ACCOUNT

# Instead of putting the time limit in your batch file,
#  submit with sbatch --time=<timelimit>
# see sbatch --help for other nice things you can add
# and also read the reference link above.

#----------------------------------------------------------

  • (Optional) If you want a motd or do any post-setups, add a file /atlasadmin/localPostUserSetup.sh. For example, Compute Canada Tier3s have this:
#
# This is a local post setup script
#   You can, for example, put in a notice for users such as the example below
#

\echo -e "
\033[1m\033[4m19 Sep 2018\033[0m                         *  *  *  *  * 
  /myScratch and /myProject are no longer available. Access it from \$HOME.
"

Known Issues

Tool Description of Issue Resolved ?
atlantis will not work in Singularity because of a java issue  
Java timezone issue affecting voms-proxy-* but only unpacked containers in Singularity Jira ticket This should now be resolved through an internal ATLASLocalRootBase setting. If not, use images; eg setupATLAS -c images+slc6.
Alternately, you can just define this environment variable: export TZ=<your local timezone>
A better option is to avoid voms-proxy-* and use arcproxy : arcproxy -r (delete), arcproxy -I (info), arcproxy -voms atlas (create)
Ganga Not yet working for backend batch submissions from inside a container Developers are working on it. But you can submit batch jobs from inside a container
Edit | Attach | Watch | Print version | History: r48 < r47 < r46 < r45 < r44 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r48 - 2019-07-11 - AsokaDeSilva
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback