Scientific Computing and Data / High Performance Computing / Documentation / Web Application: Web Server

Web Services

Web Services – General

Usage, Security/SSL, Availability and Accessibility

Getting Started with User-based Web Services

Supported Scripting Languages

Authentication

Group-based Web Services

Domain Names

Web Services – General

In General, Minerva web services is a standard Linux-Based web hosting services. Services are provided through Apache 2.4 with the MPM-ITK addon. It is designed to support basic web pages for the intent and purpose of providing web-based science portals and other applications to interact with the jobs, queues, and large science datasets on Minerva. The web services should be on-par with common unix-based web hosting services such as Dreamhost etc. Most common web frameworks and applications should easily fit into this service such as: Django, WordPress, Wikimedia, Custom Perl apps, and static content.

Server-side content is not run as the common “apache” user, but instead as the owner of that application directory. That is, each user’s content/scripts run as that user.

Account Creation Time Delay: It may take between 30min – 1hour after user account creation for the system to automatically subscribe a user to web services at https://userid.u.hpc.mssm.edu. If you are a new user, please endure the delay and check web services again before submitting a trouble ticket.

Usage, Security/SSL, Availability and Accessibility

The Minerva web services are designed to support basic web pages for the intent and purpose of providing web-based science portals and other application interactions and to interact with the jobs, queues, and large science datasets on Minerva. The web services should be on-par with common unix-based web hosting services such as Dreamhost etc. That said, we don’t recommend using web services as a primary website for a department or lab, websites with heavy traffic, or for websites that require a guaranteed amount of uptime. There are no uptime guarantees. Web services may go down during Preventative Maintenance periods. We may block IPs with suspicious activity for periods of time, possibility indefinitely, possibly automatically.
One possible usage style may be to create a website for your department or lab elsewhere and pull specific real-time generated content from the Minerva web services via javascript includes, redirects, and API’s.

Usage is unlimited for the purposes of interaction with Minerva components and viewing science.

WARNING WARNING WARNING: Be careful! Content, executables, scripts, symlinks, applications, etc within the www/ folder may be ( or are ) world accessible. Scripts and applications launched via Apache in that folder run as your user! They can access any data (including your groups’ /project data), delete data, archive data, submit jobs, cancel jobs, email people, etc as your user. You are responsible for any actions taken on your behalf!

Getting Started with User-based Web Services

Each user, by default, has a web services site. A specific user’s site is located at users.hpc.mssm.edu/~userid The Document Root for a user’s site is within their home folder in a folder called www.
Thus, https://userid.u.hpc.mssm.edu/~username/www

Step 1:
If this folder does not exist in your home directory, you should create it.

$ mkdir ~/www

Step 2:
Place content in the www folder.

$ cat > ~/www/index.html <<EOF

Hello World from my website.

EOF

Step 3:
Test your web services. Point your browser to https://userid.u.hpc.mssm.edu

Supported Scripting Languages

The following languages are supported: PHP, Python, Perl, and CGI (all others).
The Default extensions for each language:

PHP:

Web Services supports PHP. Files must end in a .php extension. The service is based on PHP 7.3.14. The following modules / patches are installed: gd ldap json mysql pear xml pear-DB pgsql mbstring pecl-imagick imap snmp suhosin zlib.
Clean URL’s may be supported through a .htaccess modification. Please read the PHP and your application’s documentation for how it should be done for your application.

An example of “Hello World” in PHP:

<?php

print “Hello World”;

?>

Python:

wsgi method:

The WSGI method of hosting python programs is new and more elegant compared to the mod_python method. It is now the standard for many frameworks, such as Django, Pyramids, etc.
( Please note this is WSGI-Script, not the daemonized / threaded style. Most apps should be able to handle this correctly. )
The Key to hosting a WSGI app is creating a directory in which you put the WSGI file, which will serve as the root of the app.
For example, if you want to host an app at https://userid.u.hpc.mssm.edu/myapp such that the URLs https://userid.u.hpc.mssm.edu/myapp/subpage and https://userid.u.hpc.mssm.edu/myapp/subpage/anotherthing are all handled by your WSGI app, then, do the following:
Create the folder by your main app name inside the root of your www folder:

mkdir ~/www/myapp

Then add a .htaccess file to declare that all files in that folder will be WSGI files. Put the following lines in a .htaccess file, like so:
Example of how to edit:

vi ~/www/myapp/.htaccess

Line to put in the file:

SetHandler wsgi-script

At this point, if you create a test WSGI app in your folder it will work, for example:

$ cat wsgione.wsgi
def application(environ, start_response):
status = '200 OK'
output = b’Hello World!’
response_headers = [(‘Content-type’, ‘text/plain’),(‘Content-Length’,
str(len(output)))]
start_response(status, response_headers)
return [output]
$




The app above would be available at https://userid.u.hpc.mssm.edu/myapp/wsgione.wsgi URLs beyond that would also be handled by it, for example: https://userid.u.hpc.mssm.edu/myapp/wsgione.wsgi/one/two/three
(In this case, of course, nothing is done with those URLs, but something could be in your app, like with django).
Of course, this URL is not very clean, so we can move the file from wsgione.wsgi to wsgione ( just removing the .wsgi ) and it will still work. Thus, you could host many apps in one folder, for example, /apps/app1, /apps/app2 .. etc etc.

That too, still may not be ideal for some use cases, so finally, it is possible to make a main top-level folder redirect into an app, however, ideally you should make a folder that only contains the one WSGI file and the .htaccess file.
You will need to call your WSGI file index.wsgi.
Then set your .htaccess file like so:

AddHandler wsgi-script .wsgi
DirectoryIndex index.wsgi
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.wsgi/$1 [QSA,PT,L]




This will rewrite https://userid.u.hpc.mssm.edu/myapp/one/two/three into the index.wsgi file and pass-in the URL parameters for the app to handle.

Note: mod_python is no longer supported in our new webserver.

Some demos on setting up your first python flask and dash app (campus network or VPN needed to access)
https://gail01.u.hpc.mssm.edu/flask_demo/
https://gail01.u.hpc.mssm.edu/dash_demo/
Code is at https://gail01.u.hpc.mssm.edu/code/

Perl and CGI:

Perl is supported through mod_perl and CGI. Simply end your file with a .pl or .cgi extension and ensure it is executable on the filesystem. (for example do a chmod +x test.pl to make it executable). As the Perl is being executed as CGI, you must put a “shebang” at the beginning of your file to point to the perl executable. The service is based on Perl 5.10.1
The following Perl modules are installed for sure, in addition to possibly others: perl-DBI perl-URI perl-IO-Socket-SSL perl-Net-DNS perl-HTML-Tagset perl-Archive-Tar perl-Net-IP perl-DBD-MySQL perl-Socket6 perl-Digest-SHA1 perl-IO-Socket-INET6 perl-Compress-Zlib perl-DBD-Pg perl-HTML-Parser perl-IO-Zlib perl-BSD-Resource perl-SGMLSpm perl-libwww-perl perl-Digest-HMAC perl-Net-SSLeay perl-Pod-POM perl-Pod-Simple perl-Date-Calc

An example Perl Hello World:

#!/usr/bin/perl
print “Content-type: text/html\\n\\n”;
print “Hello, world!\\n”;




As Perl here is executed as if it was CGI, you may instead create any executable file, including shell script (bash, csh, ksh) or even an executable binary with an appropriate name to execute it.

Authentication

It may be desirable to force visitors to authenticate before viewing a portion (or all) of your website.
Authentication is supported in two ways: System Auth or Password File

System Auth

System Auth will authenticate users using their Mt Sinai / Minerva Credentials, the same as you ssh login to Minerva. For the campus-facing web servers, password auth is supported.

To enable System Auth, you should create a .htaccess file in the root of the directory structure you wish to protect.

The .htaccess file to use for System Auth is:

AuthType Basic
AuthName Your-Site-Name
AuthBasicProvider external
AuthExternal pwauth
require valid-user




Password File

Password File will allow you to create a file with usernames and passwords combinations that will allow successful logins. Password File is not secure. It allows you to create usernames and passwords such as user “test” with password “test”. It’s not for serious authentication, however, it can be useful for light protection of folders that you just don’t want Google to accidentally run in to, for example. Please do not create bogus passwords such as test or password
To enable Password File auth, you will need to create a password file and create a .htaccess file in the root of the directory structure you wish to protect.

The password file should sit somewhere that is outside of the www/ folder. A good place may be in a folder in your home directory. If you desire, you can create a folder in your home directory for just storing these types of file, or possible a hidden folder in your home directory (one beginning with a dot)

To create a folder and password file and user named test:

$ mkdir ~/.htpasswords
$ htpasswd -c ~/.htpasswords/passdb1 test
New password:
Re-type new password:
Adding password for user test




The .htaccess file to use for Password File is:

AuthType basic
AuthName “private area”
AuthUserFile    “/hpc/users/user1/.htpasswords/passdb1”
Require            valid-user
Order allow,deny
Allow from all




Note: You should place the full and proper path to the password file in the AuthUserFile line.

General:
You may prefer some variations. For example:
Varing the line AuthName will show the user a different prompt at the login box.
Varing the require valid-user to require user user1 user2 will restrict access to user1 and user2 only. Similarly, require group group2 will restrict access to group2
Other variations are possible. Anything that may be supported in a .htaccess file is allowed (some restrictions). You can block based on remote IP etc.
Some Sites show many possible variations.

Please read the Apache documentation on Basic Auth and The Require Directive

Group-based Web Services

As described above, web services are currently only created, by default, for each user. For a group-based web service, or a custom-named web service, please submit a ticket by emailing hpchelp@hpc.mssm.edu with your request. Do note that the web service will need to run as some specific user, so please plan ahead as to which user should own the web service.
Similarly, custom web services (see below) on a per-user or per-group basis may use non-default domain names, however, it is a special request.

Domain Names

By default, the Scientific Computing group places all services below the hpc.mssm.edu domain. For automated purposes, we place the generated web service accounts below the u.hpc.mssm.edu domain (u for “users”). Although this provides a quick way for users to start working with web services, it may not be desirable a desirable URL for certain cases. Shorter domains typically look nicer. Long URL’s with usernames can be confusing.

The Scientific Computing group does not have the ability to acquire or control URL’s outside of our own domain. However, your user / group may go through either Mt Sinai IT to acquire some other *.mssm.edu (or similar) domain and/or purchase a domain from a public entity, such as Network Solutions, ENOM, or Godaddy. You will also need some sort of Domain Name Hosting services. You can then create an A or CNAME record to point to our web hosting IP address(es). We can configure your account to accept requests for these additional domains.

The Scientific Computing group and the web services do not provide Domain Name Hosting or Email Services.

SSL

SSL with domain names outside the hpc.mssm.edu domain presents some challenges. Each unique domain will require an additional IP Address. This will need to be evaluated on a case-by-case basis to see if there are remaining addresses to address the need. Please plan ahead before making any purchases to ensure the service will function correctly.
Other hybrid variations may provide useful. For example, the non-secure content can be on the custom domain and the secure content can be on the default u.hpc.mssm.edu domain. The two sites can pass the user back and forth as needed.