All use of Scientific Computing and Data resources in publications and presentations must acknowledge Mount Sinai’s CTSA funding award. Click here

Scientific Computing and Data / High Performance Computing / Documentation / File Transfer: Globus and Others

File Transfer: Globus and Others

Icahn School of Medicine at Mount Sinai (ISMMS) maintains high speed network connections with the commercial Internet and Internet2 through NYSERNet.  ISMMS currently maintains the following options for file transfer:

 

Globus
SCP, SFTP, rsync

Globus

Globus is a secure, high-performance data transfer service commonly used by researchers and institutions to move large files between systems (e.g., between your computer and a high-performance computing center, or between universities).

An advanced secure version of Globus File Manager with High Assurance HIPAA compliance BAA subscription is deployed on Minerva.

Globus Web Application

The Globus Web App provides a user-friendly interface for managing data transfers interactively, suitable for users who prefer a graphical interface over command-line operations.

Authentication with mssm.edu

To manage or share your Minerva files with collaborators, visit https://app.globus.org/ to login. When prompted for authentication, please use your Mount Sinai school email (eg, first.last@mssm.edu) for access. For more information on “Log In and Transfer Files with Globus”, click here. If you are sure you have the correct username and password (no VIP), please contact hpchelp@mssm.edu for further assistance.

Locating Minerva Collections

After login, in the File Manager tab, search Minerva in the collections. Please use the following two collections to access your files under /sc/arion and your files under home /hpc/users. Watch the owner id be89659c-xxx.

How to Share files with Globus?

Please see step by step instruction for file sharing here.

Globus CLI (Globus Command Line Interface)

The Globus CLI is a Python-based tool for interacting with Globus services from a terminal, ideal for scripting and automation. It allows users to manage file transfers, collections, and tasks. Globus CLI has been installed under the module python/3.12.5.

Authentication

Authenticate to access Globus services.

globus login

This generates a URL. Copy it to a browser, log in with your mssm.edu credentials, and consent to CLI access. Paste the resulting authorization code back into the terminal.

Use globus whoami to verify your identity.

Log out with globus logout when done.

Locating Minerva Collections

Search for Minerva collections.

In the provided Globus endpoint search results for “Minerva Arion”, the UUID refers to the ID column, which uniquely identifies the endpoint. In this case, the UUID is:

Save collection Universally Unique Identifiers (UUIDs) as environment variables for ease.

6621ca70-103f-4670-a5a7-a7d74d7efbb7

export ENDPOINT1=uuid1
export ENDPOINT2=uuid2

View directory contents on a collection

globus ls $ENDPOINT1:/path/to/directory

Create a directory.

globus mkdir $ENDPOINT1:/path/to/new_directory

Data Transfer

Transfer files or directories between collections.

globus transfer $ENDPOINT1:/path/to/source $ENDPOINT2:/path/to/destination --recursive

Use --recursive for directories, --preserve-mtime to maintain timestamps.
Transfer multiple files using a text file.

echo "/source/file1.txt /dest/file1.txt" > batch.txt
echo "/source/file2.txt /dest/file2.txt" >> batch.txt
globus transfer $ENDPOINT1 $ENDPOINT2 --batch batch.txt
Task ID: abc12345-6789-0123-4567-890123456789

Here, abc12345-6789-0123-4567-890123456789 is the <task_id>.

Check transfer status.

globus task show <task_id>

For more information on Globus CLI, please review the official documentation https://docs.globus.org/cli/.

Globus API (Application Programming Interface)

The Globus API enables programmatic access to Globus services, ideal for integrating data transfer into custom workflows or applications. It uses RESTful endpoints and OAuth 2.0 for authentication.

For detailed information on the use of Globus API, please review the official documentation https://docs.globus.org/api/.

Globus Transfer Protocols and HTTPS Server-Side Size Limit

Globus utilizes GridFTP for high-performance and reliable data transfers between endpoints. GridFTP is an extension of the standard FTP protocol optimized for high-bandwidth, wide-area networks, providing features like fault recovery and automated performance tuning.

In addition to GridFTP, Globus also supports:

  • UDT: Another protocol used for managed file transfer
  • HTTPS: For direct uploads and downloads to/from collections using a web browser, particularly for smaller datasets.
General Protocol Association by Globus Application
  1. Globus Web Application
    • GridFTP: Used for asynchronous bulk file transfers between endpoints (for example, mapped/guest collections). Initiated via the File Manager when transferring files between endpoints/collections.
    • HTTPS: Used for synchronous operations like direct file uploads/downloads in the File Manager or accessing file previews.
    • UDT: Used transparently for transfers when endpoints are configured to support it (less common, typically for high-performance server endpoints).
  2. Globus CLI
    • GridFTP: Primary protocol for file transfer commands like globus transfer, globus ls --recursive, or globus delete.
    • HTTPS: Used for authentication (OAuth flows) and some metadata operations (for example, globus endpoint show).
    • UDT: Used transparently with GridFTP commands if the endpoint supports UDT.

The Web App’s direct upload/download feature is designed for convenience and smaller files, not bulk transfers. For larger datasets, Globus encourages asynchronous transfers using GridFTP, which is more robust. 

A 16 GB limit for HTTPS transfers has been set on Minerva to prevent server overload.

For direct data transfer between your local device and Minerva, instead of the Web App’s direct upload/download feature, it is recommended using Globus Connect Personal (GCP). GCP runs on your personal device (e.g., laptop, desktop), turning it into a Globus endpoint. This allows you to initiate transfers using GridFTP between your device and other Globus endpoints (for example, Globus Connect Server).

Globus Connect Personal

Globus Connect Personal is a lightweight application that turns your personal computer (such as a laptop or desktop) into a Globus endpoint, allowing you to securely transfer files to and from it using the Globus data transfer service.

Users handling HIPAA/sensitive data on machines running Globus Connect Personal, please check High Assurance in the preference. Please ask hpchelp@hpc.mssm.edu to add you to the Globus BAA group subscription.

 

Globus Subscription/Globus Plus

Globus subscription (formerlly known as Globus Plus) enables users to share data and transfer files between Globus Connect Personal endpoints (such as, personal computers) and to share data from those endpoints. If you need to upgrade your Globus account to Globus Plus, please contact us at hpchelp@hpc.mssm.edu to request a Globus Plus invite.

SCP, SFTP, rsync:

The standard transfer utilities, SCP, SFTP and rsync can be used to transfer files to and from MSSM systems. These utilities are usually already installed on Linux/Unix machines, and Mac’s. There are also many command and graphical clients available. Due to familiarity and ease, these may be the best choice for transferring scripts and small files, however, these options can be slow in comparison, and may be ill suited for transferring large amounts of data, such as hundreds of TB’s. More information on these utilities can be found on the transfer utilities page.

For windows, the best application for SCP is “PSCP”. It is a command-line tool which replicates the Linux / Unix tools. It is very fast and efficient. You can acquire PSCP here.

A graphical alternative for Windows and Mac is CyberDuck. It is also quick, efficient, and full-featured.

Note: Ensure the Reuse password feature of CyberDuck is disabled. CyberDuck will try to reuse your one-use six-digit VIP token code repeatedly until you get locked out!

Unsupported Methods:

Minerva currently does not support FTP or BBCP. Please utilize one of the above methods.