[et_pb_section fb_built=”1″ fullwidth=”on” admin_label=”Section” _builder_version=”4.9.0″ _module_preset=”default”][et_pb_fullwidth_menu menu_id=”15″ menu_style=”centered” fullwidth_menu=”on” active_link_color=”#d80b8c” dropdown_menu_line_color=”#221f72″ _builder_version=”4.9.0″ _module_preset=”default” menu_font=”|600|||||||” menu_text_color=”#FFFFFF” menu_font_size=”16px” background_color=”#221f72″ background_layout=”dark” sticky_position=”top”][\/et_pb_fullwidth_menu][\/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.9.0″ custom_padding=”0px||0px||false|false”][et_pb_row _builder_version=”4.9.0″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_padding=”||0px||false|false”][et_pb_column type=”4_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_text admin_label=”Breadcrumb” _builder_version=”4.9.0″ _module_preset=”default”]<\/p>\n
Scientific Computing and Data<\/a> \/ High Performance Computing<\/a> \/ Documentation<\/a> \/ Long-Term Data Archive: TSM<\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=”4.9.0″ _module_preset=”default”][et_pb_column type=”4_4″ _builder_version=”4.9.0″ _module_preset=”default”][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”]<\/p>\n Important data stored on Minerva can be protected by archiving the data in the IBM Tivoli Storage Management (TSM) system (renamed as Spectrum Protect after v8.1.21.0) installed on a Minerva server.<\/p>\n The TSM system will create two long-term tape copies of your data. One copy will be stored in the IBM tape library that is part of the Minerva complex and be available for rapid recall; the second copy will be stored off-site in a secure data vault for disaster protection. Data on both copies will be encrypted to deter unauthorized access.<\/p>\n <\/span><\/p>\n \u00a0<\/span><\/p>\n Please review the\u00a0Minerva TSM Archival Policy V1 before your TSM activities click here<\/a><\/span><\/p>\n <\/p>\n The archived data will have a retention time of 6 years and will then be deleted, please check the expiration date of your archived files.\u00a0 This is the responsibility of the user!<\/strong><\/p>\n <\/p>\n TSM client or Spectrum Protect client v8.1.21.0 is installed on all login nodes.<\/p>\n In the TSM system, archived data is organized by nodes. A node is a logical abstraction and may correspond to various physical entities. On Minerva, each user is treated as a separate node, and the node identity is defined by the user’s ID (userid).<\/p>\n The TSM system can be accessed via either a GUI or the command line. The command line mode is particularly useful when archiving large datasets in that it can be issued using the screen command. The screen can then be detached and the command can run unattended for the hours it may take to archive the data.<\/p>\n Access TSM with GUI<\/a> General suggestions<\/strong><\/span><\/p>\n Tar small files before archive.<\/strong><\/p>\n Because all the files that are archived are entered into a database, to prevent overflowing this database we ask that you first use the tar command to create a unix tar archive of bundles of small files and then archive the tar file to TSM.\u00a0For information about tar, see the man pages (man tar) on Minerva or check out this\u00a0link<\/a>.<\/p>\n Command Line and Screen is recommended for large data archiving or retrieval.<\/strong><\/p>\n Using the graphical interface for large data archiving or retrieval is not recommended, as it requires keeping the interactive session open until the process completes. Instead, it is advisable to start a Long retrieval time is expected.<\/strong><\/p>\n Due to the large volume of archived data and limited tape slots in the TSM library, most tapes are stored in cabinets rather than loaded in the library. When you issue a retrieval request, our operators are notified by email and will manually locate and load the requested tape into the library. This manual process typically takes up to 1.5 hours, during which the request may appear as “[ – ]” with no visible progress.<\/p>\n Once the tape is loaded, the TSM library will automatically mount it and begin data transfer, which proceeds at a reasonably fast rate.<\/p>\n Please note that tape check-in errors may occur when multiple retrieval requests are processed simultaneously. If you receive an error such as “data is unavailable,” submit a support ticket and we\u2019ll assist you promptly.<\/p>\n Warning: If you choose to automatically delete files after archiving and later delete the corresponding archive object from TSM, the data will be permanently lost and cannot be recovered. Use this option with extreme caution.<\/strong><\/p>\n <\/p>\n To access the system via the X-Windows GUI, enter the dsmj<\/strong> command.\u00a0 dsmj<\/strong> requires java, so the java module will need to be loaded:<\/p>\n The following window will be displayed:<\/p>\n [\/et_pb_text][et_pb_image src=”https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/tsm\/tsm_gui.png” align=”center” _builder_version=”4.9.0″ _module_preset=”default” width=”60%” max_width=”60%” module_alignment=”center” custom_margin=”10px||20px||false|false” custom_padding=”10px||20px||false|false”][\/et_pb_image][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”10px||||false|false” custom_padding=”10px||||false|false”]<\/p>\n We are not supporting backup so ignore the top panel of the display.\u00a0 If you click on “Archive<\/strong>“, the following screen will appear.\u00a0Files are displayed in the standard descending tree format and you can drill down to any sub-directory or file.\u00a0 When you get to the last directory, you click on the “folder<\/strong>” icon to expand the list of files.<\/p>\n <\/p>\n [\/et_pb_text][et_pb_image src=”https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/tsm\/tsm_gui_archive.png” align=”center” _builder_version=”4.9.0″ _module_preset=”default” width=”80%” max_width=”80%” module_alignment=”center”][\/et_pb_image][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”20px||20px||false|false” custom_padding=”20px||20px||false|false”]<\/p>\n To select files to be archived, click the box nest to the file name.\u00a0 To select the entire sub-directory, click on the box next to the sub-directory you wish to archive.\u00a0 (the boxes next to each file will automatically be filled in).<\/p>\n You can set the “description<\/strong>” of the archive package by entering your description into the text box at the top of the screen.<\/p>\n Once files are selected and you have annotated your archive by means of the “description<\/strong>” text box, click on the “archive<\/strong>” button and the operation will commence.<\/p>\n [\/et_pb_text][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”]<\/p>\n To retrieve files, the operation is the same except that you would click on Retrieve<\/strong>. Note that instead of the hierarchical display being by file, the display will be by “description<\/strong>“.\u00a0 Drilling down on one of the archive packages will display the file structure of what was archived.<\/p>\n <\/p>\n [\/et_pb_text][et_pb_image src=”https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/tsm\/tsm_gui_retrieve.png” align=”center” _builder_version=”4.9.0″ _module_preset=”default” width=”80%” max_width=”80%” module_alignment=”center”][\/et_pb_image][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”20px||20px||false|false” custom_padding=”20px||20px||false|false”]Select the files and click the “Options<\/strong>” button to select the options you want for the Retrieve.[\/et_pb_text][et_pb_image src=”https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/tsm\/tsm_gui_retrieve_options.png” align=”center” _builder_version=”4.9.0″ _module_preset=”default” width=”80%” max_width=”80%” module_alignment=”center”][\/et_pb_image][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”20px||20px||false|false” custom_padding=”20px||20px||false|false”]<\/p>\n Then click “retrieve<\/strong>” to continue.<\/p>\n [\/et_pb_text][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”10px||20px||false|false” custom_padding=”10px||20px||false|false”]<\/p>\n From the main screen, select Utilities > Delete Archive Data:[\/et_pb_text][et_pb_image src=”https:\/\/scpublic.dmz.hpc.mssm.edu\/HPC\/tsm\/tsm_delete.png” align=”center” _builder_version=”4.9.0″ _module_preset=”default” width=”60%” max_width=”60%” module_alignment=”center” custom_margin=”10px||20px||false|false” custom_padding=”10px||20px||false|false”][\/et_pb_image][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”10px||20px||false|false” custom_padding=”10px||20px||false|false”]<\/p>\n <\/p>\n The hierarchical display of the archive data will appear as in Retrieve, but the option will be only to delete.<\/p>\n [\/et_pb_text][et_pb_text _builder_version=”4.9.0″ header_font=”|600|||||||” header_text_color=”#221f72″ header_2_text_color=”#221f62″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_margin=”10px||20px||false|false” custom_padding=”10px||20px||false|false”]<\/p>\n The command to access the TSM system is:\u00a0 You can either issue the command through your login shell, for example:<\/p>\n or<\/strong> you can launch the spectrum protect (TSM) shell and issue the command there:<\/p>\n Caution:<\/strong><\/span><\/em> Example:<\/span><\/em> If one is in their home directory and specifies the file name as Caution:<\/strong><\/span><\/em> This makes a difference when you ask the The various commands to the system are introduced below by way of examples for\u00a0<\/strong><\/span>screen<\/span><\/span>,<\/span>\u00a0<\/a>archive<\/span><\/span>,<\/span><\/a>\u00a0query<\/a><\/span><\/span>,\u00a0retrieve<\/a><\/span><\/span>,\u00a0delete<\/span><\/span><\/a><\/p>\n <\/p>\n <\/a>Screen Commands<\/span><\/strong><\/p>\n The\u00a0 Some simple, elementary examples:<\/p>\n After starting a new screen session, you can run the desired command\u2014whether it begins executing immediately or is momentarily blocked by TSM while waiting for resources. To detach from the session and let the process continue in the background, press Alternatively, you can start a screen session by simply typing screen, which opens an interactive shell. From there, you might run a few setup commands before launching Long-Term Data Archive: TSM<\/h1>\n
Contents<\/span><\/b>
Archiving Data<\/a>
Access TSM with GUI<\/a>
Access TSM with Command Line<\/a>
Access Archived Files<\/a>
Frequently Asked Questions<\/a><\/h3>\n<\/a>Archiving Data<\/h1>\n
Minerva Archival Storage Policy <\/strong><\/h1>\n
Data Retention Policy<\/span><\/h2>\n
How to Access TSM<\/span><\/h2>\n
Access TSM with Command Line (recommended)<\/a><\/p>\nscreen<\/code> session and use command-line tools to perform the archiving. This allows you to safely detach from the session while the operation continues in the background.<\/p>\n<\/a>Access TSM with GUI<\/h1>\n
\n
$ module load java\r\n $ dsmj -se=userid<\/span>\r\n<\/code><\/pre>\n<\/blockquote>\nArchive by GUI<\/span><\/h2>\n
Retrieval by GUI<\/span><\/h2>\n
Delete Archive by GUI<\/span><\/h2>\n
<\/a>Access TSM with Command Line<\/h2>\n
dsmc<\/code>.\u00a0 The option needed to specify your node name is:\u00a0-se=userid<\/code>. You must always indicate your node (user name) on the command line or the command will fail.<\/p>\n\n
$ dsmc q archive -se= userid \/home\/proj1\/ -sub=yes <\/code><\/pre>\n<\/blockquote>\n\n
$ dsmc -se= userid \r\nProtect> q archive \/home\/proj1\/ -sub=yes \r\n<\/code><\/pre>\n<\/blockquote>\n
By default, the dsmc<\/code> command operates only within the current file system. For example, if you are in the home file system (\/hpc\/users<\/code>), wildcard patterns will not match files in the scratch file system, and vice versa. To access files in a different file system, you must explicitly specify the full path to the desired directory.<\/p>\n
Assume these files exist:<\/p>\n\/hpc\/users\/minerva_user\/a.file<\/code>
\/hpc\/users\/minerva_user\/b.file<\/code><\/p>\n\/sc\/arion\/scratch\/minerva_user\/c.file<\/code>
\/sc\/arion\/scratch\/minerva_user\/d.file<\/code><\/p>\n*<\/code>, only file a.file<\/code> and b.file<\/code> will be seen.
If one is in the scratch directory and specifies the file name as \"\/hpc\/users\/minerva_user\/*\"<\/code>, again, only a.file<\/code> and b.file<\/code> will be seen.<\/p>\n
TSM distinguishes between treating a directory file as a directory or treating it as a file. When a file is archived, the parent directory file is also archived.
In general, if a directory is to be considered a directory, then a slash should follow the name.\u00a0 E.g.:<\/p>\n\/hpc\/users\/m_user\/all_my_data<\/code> is the file all_my_data.\u00a0 Whether it is a directory or not it will be treated as a simple file.
\/hpc\/users\/m_user\/all_my_data\/<\/code> is the directory all_my_data and all the files in the directory.<\/p>\ndsmc<\/code> commands to work with all the files below the directory with the -sub<\/code> option. In the case with no slash, -sub<\/code> has no effect since you are talking about the file; with the slash, -sub<\/code> will act on the files below the directory name.\u00a0<\/strong>See example below<\/span><\/span>.<\/a><\/p>\nscreen<\/code>\u00a0command is particularly useful here when archiving large files. For information on how to use the screen command check the man page (man screen<\/code>).<\/p>\n\n
screen dsmc archive -se=userid<\/em> big_directory -sub=yes -description=\"Irreplaceable data 1 Apr 2012\" <\/code><\/pre>\n<\/blockquote>\nCtrl-a<\/code> followed by d<\/code>. You can then continue working or log out entirely.<\/p>\ndsmc<\/code>. As before, press Ctrl-a d<\/code> to detach the session.<\/p>\n