Using on-the-fly RStudio/Jupyter Web on Minerva HPC

We have developed simple tool scripts for users to launch RStudio server/Jupyter Notebook with interactive web interface inside the LSF job on the terminal with one command.

These scripts are located in the login nodes under

/usr/local/bin/{minerva-rstudio-web.sh, minerva-rstudio-web-r4.sh, minerva-jupyter-web.sh, minerva-jupyter-r-web.sh, minerva-jupyter-module-web.sh}

The

minerva-jupyter-web.sh

will launch a simple jupyter notebook session with GPU support. The

minerva-jupyter-r-web.sh

supports both Jupyter notebook Python 3 and R3 kernels. The

minerva-jupyter-module-web.sh

will use the installed modules to start Jupyter sessions. The

minerva-rstudio-web.sh

will start RStudio web with R3 and

minerva-rstudio-web-r4.sh

will start RStudio with R4 kernel.

Usage:

For example, to start an RStudio web session with R, on the login nodes, run commands minerva-rstudio-web-r4.sh (using R 4.0.3) or minerva-rstudio-web.sh (R 3.6.2) with default resource configuration and URL to access it. Please see the -h option for help messages containing resource requests and installing packages.

Example:

$ sh minerva-rstudio-web-r4.sh 
[INFO] Image not specified, check if previously used
[INFO] Found previously used image 
/hpc/users/guow03/minerva_jobs/rstudio_jobs/singularity-rstudio.simg. 
Using it.
[INFO] Project is not specified, or is acc_null, 
using 1st avail project.
[INFO] Project to use is acc_hpcstaff
[INFO] Parameters used are: 
[INFO] -n    4
[INFO] -M    3000
[INFO] -W    6:00
[INFO] -P    acc_hpcstaff
[INFO] -J    rstudio
[INFO] -q    premium
[INFO] -R    null
[INFO] -i    /hpc/users/guow03/minerva_jobs/rstudio_jobs/
singularity-rstudio.simg
[INFO] Submitting rstudio job...
Job <25962439> is submitted to queue .
[INFO] See below for web access when job starts.
Job <25962439> : Not yet started.
[INFO] Job is pending
Job <25962439> : Not yet started.
[INFO] Job is pending
Job <25962439> : Not yet started.
[INFO] Job is pending
Job <25962439> : Not yet started.
[INFO] Job is pending
[INFO] Job is running, wait for link
[INFO] Job is running, wait for link
[INFO] Job is running, wait for link
<< output from stdout >>
Using local available port 8787
Using password in /hpc/users/guow03/minerva_jobs/rstudio_jobs/
.rstudio_onthefly_password
RStudio started in the singularity container with PID 306586.
Making sure it is alive
Checking 3, next check in 5 seconds.
   PID TTY          TIME CMD
306586 ?        00:00:00 singularity
Checking 2, next check in 5 seconds.
   PID TTY          TIME CMD
306586 ?        00:00:00 starter-suid
Checking 1, next check in 5 seconds.
   PID TTY          TIME CMD
306586 ?        00:00:00 starter-suid

SSH port forwarding to 10.95.46.103 with PID 306725.

Rstudio is started on compute node lc03e29, port 8787

Access the RStudio Web using your web browser:  
http://10.95.46.103:52439 

<< output from stderr >>

The RStudio web interface will be available at URL http://10.95.46.103:52439. Use the browser on your laptop to access it. Here 10.95.46.103 is the login node IP on the campus network and 52439 is the port. 2439 is the last 4 digit of the job 25962439 running the instance.

For Jupyter sessions, you will see links blinking with the url with token (no password), like the following.

<pre>

[INFO] Copy the following link in your browser for the jupyter notebook web access.

[INFO] http://10.95.46.103:42206/?token=f8e4e879f6ec495ed706f5269a328b7270c956c167534499

</pre>

Copy the above url and paste it in your browser to access the Jupyter Notebook web session.

What happens behind the scene? This tool wraps the following tasks in one command.

1. downloads a custom built Singularity container image of RStudio in your home directory
2. prompts and creates a password for the RStudio interface,
3. writes and submits an LSF job script to launch the RStudio within the image,
4. provides the URL link to access the instance

Currently, the minerva-rstudio-web-r4.sh script is the preferred version as it is built with common libraries including "DBI", "odbc", "shiny", "devtools", "ggplot2", "tidyverse", "tidymodels", "car", "dplyr", "tidyr". If you want to switch the scripts to use, move or rename the downloaded image singularity-rstudio.simg in $HOME/minerva_jobs/rstudio_jobs.

See the Package Installations section in the help message for packages are not installed.

Common problems:
1. When a user runs an rstudio session on a compute node, there will be two lock directories created in the /tmp directory: /tmp/rstudio-session and /tmp/rstudio-server. These directories blocks other users to run more sessions since the new comer will not have permission to overwrite the directories. The script will error out in the output. Workaround is to run the script again till the job is dispatched to a node with no existing session.

2. Plotting in rstudio errors due to X11 display not working when using the v4 version. This problems has been fixed with updated image. Please remove your old image and run the script.