Scientific Computing and Data / High Performance Computing / Documentation / Conda

Conda

Contents
Usage
Using Conda with Python
Using Conda with R

 
Conda is an open-source package management system and environment management system that runs on Windows, macOS, Linux and z/OS. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language.

The conda package and environment manager is included in all versions of Anaconda and Miniconda. Many versions of Anaconda are available to use on Minerva. To see a list of installed versions of Anaconda on the cluster, use Lmod’s spider command:

$ ml spider anaconda

---------------------------------------------------------------------------- 
  anaconda2: 
---------------------------------------------------------------------------- 
     Versions: 
        anaconda2/latest 
        anaconda2/2019.03 
        anaconda2/2019.10dev 
        anaconda2/2019.10 
 
 
---------------------------------------------------------------------------- 
  For detailed information about a specific "anaconda2" module (including how to load the modules) use the module's full name. 
  For example: 
 
 
     $ module spider anaconda2/latest 
---------------------------------------------------------------------------- 
 
 
---------------------------------------------------------------------------- 
  anaconda3: 
---------------------------------------------------------------------------- 
     Versions: 
        anaconda3/latest 
        anaconda3/4.5.11 
        anaconda3/4.6.4 
        anaconda3/2018.12 
        anaconda3/2019.10 
        anaconda3/2020.11 
        anaconda3/2021.5 
 
 
---------------------------------------------------------------------------- 
  For detailed information about a specific "anaconda3" module (including how to load the modules) use the module's full name. 
  For example: 
 
 
     $ module spider anaconda3/4.6.4 
----------------------------------------------------------------------------

 

Usage

When creating an environment, conda will download large amounts of installation tarballs to the package cache directory. The default is “~/.conda/pkgs” in /home directory, however on Minerva, user’s home directory has 20 GB quota. Therefore, instead of using this default setting, it is recommended to redirect the downloaded package cache directory to a different location, which can be specified in the “~/.condarc” file. Below steps listed steps for redirecting the package cache location:

Edit conda user config file: “~/.condarc” to redirect the package cache and env directory. For example, after opening “~/.condarc”, add the following lines:

envs_dirs: 
- /sc/arion/work/your_username/test-env/envs 
pkgs_dirs: 
- /sc/arion/work/your_username/test-env/pkgs

If you do not have a “~/.condarc” file, use the command below to create one:

$ touch ~/.condarc

Before trying to create a conda virtual environment, do a module purge to clean up the current environment:

$ module purge

Then load an Anaconda module. For example:

$ module load anaconda3/2021.5

Check conda information. If conda is available, you will see:

$ conda info 
… 
       conda version : 4.11.0 
 conda-build version : 3.21.4 
      python version : 3.8.8.final.0 
    virtual packages : __linux=3.10.0=0 
... 
       package cache : /sc/arion/work/your_username /pkgs 
    envs directories : /sc/arion/work/your_username /envs 
                       /hpc/users/your_username/.conda/envs 
                       /hpc/packages/minerva-centos7/anaconda3/2021.5/envs 
            platform : linux-64 
...

Also make sure the envs and package cache directories have been redirected to the correct directory.
Next is to create a conda virtual environment. For example, creating a conda environment called “test” by:

$ conda create -n test

The conda environment should then be activated before installing packages:

$ source activate test

The name of the virtual environment will also show in front of your username in the prompt:

(test) [user_name@li03c03 ~]$

Then you can install packages using the command:

$ conda install -c channel_name package1=version package2=version

Channel name and package version are optional.
If you want to leave your virtual environment, simply run:

$ conda deactivate

Other useful conda commands:
Checking current and available conda environment:

$ conda info –-envs

Check installed packages:

$ conda list

Upgrade and uninstall packages inside a conda environment:

$ conda upgrade/update package1 package2  
$ conda uninstall/remove package1 package2

Remove a conda environment:

$ conda env remove -n env_name

Using Conda with Python

Make sure to unset PYTHONPATH and PYTHONHOME in case they are specified previously:

$ unset PYTHONPATH 
$ unset PYTHONHOME

The version of Python in the conda environment can be specified when creating the conda environment, for example, specify python 3.6 to be used in the environment “test”:

$ conda create –n test python=3.6

Or install python after creating the conda environment:

$ conda install python=3.6 

Using Conda with R

Make sure to unset R_LIBS and R_LIBS_USER in case they are specified previously:

$ unset R_LIBS 
$ unset R_LIBS_USER 

Also, you might need to comment lines related to these variables in .Renviron and .Rprofile file in your home directory.
The version of R in the conda environment can be specified when creating the conda environment, for example, specify R 4.2.0 to be used in the environment “test”:

$ conda create –n test r-base=4.2.0 

Or install R after creating the conda environment:

$ conda install r-base=4.2.0 

When using conda to install R packages, you will need to add r- before the regular package name. For instance, if you want to install rbokeh, you will need to use:

$ conda install r-rbokeh  

or for rJava, type:

$ conda install r-rjava 

For more information about using conda, please click here.