[et_pb_section fb_built=”1″ fullwidth=”on” _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_fullwidth_menu menu_id=”15″ menu_style=”centered” fullwidth_menu=”on” active_link_color=”#d80b8c” dropdown_menu_line_color=”#221f72″ _builder_version=”4.16″ _module_preset=”default” menu_font=”|600|||||||” menu_text_color=”#FFFFFF” menu_font_size=”16px” background_color=”#221f72″ background_layout=”dark” sticky_position=”top” global_colors_info=”{}”][\/et_pb_fullwidth_menu][\/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.16″ custom_padding=”0px||0px||false|false” global_colors_info=”{}”][et_pb_row _builder_version=”4.16″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_padding=”||0px||false|false” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ custom_padding=”|||” global_colors_info=”{}” custom_padding__hover=”|||”][et_pb_text admin_label=”Breadcrumb” _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”]<\/p>\n
Scientific Computing and Data<\/a> \/ High Performance Computing<\/a> \/ Documentation<\/a> \/ Python and Jupyter Notebook<\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_row _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_text admin_label=”Python” _builder_version=”4.16″ _module_preset=”default” header_font=”|600|||||||” header_text_color=”#221f72″ header_font_size=”26px” header_2_text_color=”#221f72″ header_2_font_size=”24px” global_colors_info=”{}”]<\/p>\n <\/p>\n <\/p>\n Python is an interpreted programming language that has become increasingly popular in high-performance computing environments because it\u2019s available with an assortment of numerical and scientific computing libraries (numpy, scipy, pandas, etc.), relatively easy to learn, open source, and free.<\/p>\n Many versions of Python are available to use on Minerva. To see a list of installed versions of Python on the cluster, use Lmod\u2019s spider command:<\/p>\n [\/et_pb_text][et_pb_image src=”https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/python\/python_modules.jpg” align=”center” admin_label=”Python Modules Image” _builder_version=”4.16″ _module_preset=”default” width=”60%” max_width=”60%” global_colors_info=”{}”][\/et_pb_image][et_pb_text admin_label=”Usage” _builder_version=”4.16″ _module_preset=”default” header_font=”|600|||||||” header_text_color=”#221f72″ header_font_size=”26px” header_2_text_color=”#221f72″ header_2_font_size=”24px” global_colors_info=”{}”]<\/p>\n Python may be run either interactively, or as a batch process reading commands from a script file. To run Python interactively, open an interactive session by submitting a job to the interactive LSF queue. Then simply execute the Python command. After the terminal is launched in the interactive mode, users can run Python commands at the prompt:<\/p>\n <\/p>\n [\/et_pb_text][et_pb_image src=”https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/python\/python_usage.jpg” align=”center” admin_label=”Python Usage Image” _builder_version=”4.16″ _module_preset=”default” width=”60%” max_width=”60%” global_colors_info=”{}”][\/et_pb_image][et_pb_text admin_label=”Install Packages Locally” _builder_version=”4.16″ _module_preset=”default” header_font=”|600|||||||” header_text_color=”#221f72″ header_font_size=”26px” header_2_text_color=”#221f72″ header_2_font_size=”24px” global_colors_info=”{}”]<\/p>\n <\/p>\n There are multiple ways to install and manage Python packages locally. Here we present several of them in detail below as examples:\u00a0<\/p>\n [\/et_pb_text][et_pb_text admin_label=”Anaconda” _builder_version=”4.27.4″ _module_preset=”default” header_font=”|600|||||||” header_text_color=”#221f72″ header_font_size=”26px” header_2_text_color=”#221f72″ header_2_font_size=”24px” custom_margin=”20px||||false|false” global_colors_info=”{}”]<\/p>\n You can also create a virtual environment using Conda in miniforge3 modules. Please be familiar with official Conda user guide<\/a> and our documentation<\/a> for more information about using Conda.<\/p>\n [\/et_pb_text][et_pb_image src=”https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/conda\/miniforge.png” align=”center” admin_label=”Anaconda3 Image” _builder_version=”4.27.4″ _module_preset=”default” width=”60%” max_width=”60%” custom_margin=”10px||20px||false|false” custom_padding=”10px||20px||false|false” global_colors_info=”{}”][\/et_pb_image][et_pb_text admin_label=”Venv” _builder_version=”4.27.4″ _module_preset=”default” header_font=”|600|||||||” header_text_color=”#221f72″ header_font_size=”26px” header_2_text_color=”#221f72″ header_2_font_size=”24px” global_colors_info=”{}”]<\/p>\n venv<\/a> allows you to manage separate package installations for different projects. They essentially allow you to create a \u201cvirtual\u201d isolated Python installation and install packages into that virtual installation. When you switch projects, you can simply create a new virtual environment and not have to worry about breaking the packages installed in the other environments. It is always recommended to use a virtual environment while developing Python applications.<\/p>\n Python will need to be loaded in module before using venv:<\/p>\n Creation of virtual environments is done by executing the command venv:<\/p>\n venv will create a virtual Python installation in the \u201c\/path\/to\/new\/virtual\/env\u201d folder. Activating a virtual environment will put the virtual environment-specific python and pip executables into your shell\u2019s PATH. It should be in the env directory:<\/p>\n The name of the virtual environment will also show in front of your username in the prompt:<\/p>\n As long as your virtual environment is activated, you can use pip for package installation and pip will install packages into that specific environment by default. You\u2019ll also be able to import and use packages in your Python application. If you want to re-enter the virtual environment just follow the same instructions above about activating a virtual environment. There\u2019s no need to re-create the virtual environment.[\/et_pb_text][et_pb_text admin_label=”Pip” _builder_version=”4.16″ _module_preset=”default” header_font=”|600|||||||” header_text_color=”#221f72″ header_font_size=”26px” header_2_text_color=”#221f72″ header_2_font_size=”24px” custom_padding=”10px||||false|false” global_colors_info=”{}”]<\/p>\n Pip is the package installer for Python<\/a>. You can use it to install packages from the Python Package Index<\/a> Python will need to be loaded in module before using pip:<\/p>\n The syntax of installing a single Python package is:<\/p>\n For example,<\/p>\n Packages will be installed in:<\/p>\n For example, for Python 3.12.5, the path is:<\/p>\n Then, prepend the package path and bin path to PYTHONPATH and PATH environment variables:<\/p>\n You should be able to use the new package now.<\/p>\n You can also install packages to a specific location by adding the –prefix:<\/p>\n You will also need to prepend the paths as shown above:<\/p>\n [\/et_pb_text][et_pb_text admin_label=”Jupyter” _builder_version=”4.27.4″ _module_preset=”default” header_font=”|600|||||||” header_text_color=”#221f72″ header_font_size=”26px” header_2_text_color=”#221f72″ header_2_font_size=”24px” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″]<\/p>\n <\/p>\nPython and Jupyter Notebook<\/h1>\n
Contents<\/strong>
Python<\/a>
Jupyter Notebook<\/a><\/span><\/h2>\n<\/a>Python<\/h2>\n
Usage<\/h3>\n
Installing New Packages Locally<\/h3>\n
\n
\n
\n
$ ml python<\/code><\/pre>\n<\/blockquote>\n\n
$ python -m venv \/path\/to\/new\/virtual\/env<\/strong><\/code><\/pre>\n<\/blockquote>\n
\nBefore you can start installing or using packages in your virtual environment, you\u2019ll need to activate it:<\/p>\n\n
$ unset PYTHONPATH \n$ source \/path\/to\/new\/virtual\/env\/bin\/activate<\/strong><\/code><\/pre>\n<\/blockquote>\n
\nYou can confirm you\u2019re in the virtual environment by checking the location of your Python interpreter:<\/p>\n\n
$ which python<\/code><\/pre>\n<\/blockquote>\n\n
\/path\/to\/new\/virtual\/env\/bin\/python<\/code><\/pre>\n<\/blockquote>\n\n
(env<\/strong>) [user_name<\/strong>@li04e01 ~]$<\/code><\/pre>\n<\/blockquote>\n
\nIf you want to switch projects or otherwise leave your virtual environment, simply run:<\/p>\n\n
$ deactivate<\/code><\/pre>\n<\/blockquote>\n\n
and other indexes.<\/p>\n\n
$ ml python<\/code><\/pre>\n<\/blockquote>\n\n
$ pip install --user package_name==version<\/code><\/pre>\n<\/blockquote>\n\n
$ pip install --user numpy==1.21.6<\/code><\/pre>\n<\/blockquote>\n\n
~\/.local\/lib\/python_version\/site-packages\/<\/code><\/pre>\n<\/blockquote>\n\n
~\/.local\/lib\/python3.12\/site-packages\/<\/code><\/pre>\n<\/blockquote>\n\n
$ export PYTHONPATH=~\/.local\/lib\/python_version\/site-packages\/:$PYTHONPATH\r\n>>> $ export PATH=~\/.local\/bin\/:$PATH<\/code><\/pre>\n<\/blockquote>\n\n
$ python \r\n>>> import numpy \r\n>>> numpy.__version__ \r\n'1.21.6'<\/code><\/pre>\n<\/blockquote>\n\n
$ pip install --prefix=\/path\/to\/folder package_name==version<\/code><\/pre>\n<\/blockquote>\n\n
$ export PATH=\/path\/to\/folder\/bin\/:$PATH \r\n$ export PYTHONPATH=\/path\/to\/folder\/lib\/python_version\/site-packages\/:$PYTHONPATH<\/code><\/pre>\n<\/blockquote>\n<\/a>Jupyter Notebook<\/h2>\n