\nbchkpnt [-k] jobid<\/p>\n<\/blockquote>\n
Note: The optional -k<\/b> designates that the job is to be killed after the checkpoint is taken. Because file location information is not saved or used on restart, it is highly recommended that the -k option be specified.<\/i><\/p>\n
To restart the job, one uses the brestart<\/b> command:<\/p>\n\nbrestart [options] checkpointFolder jobid<\/p>\n<\/blockquote>\n
Example: brestart -W 4:00 chkpnt 193876<\/strong>
An LSF script example:<\/p>\n\n#BSUB -q test
#BSUB -P acc_hpcstaff
#BSUB -W 2:00
#BSUB -n 1
#BSUB -k \u201cchkpnt method=criu\u201d
#BSUB -oo %J.out
\/hpc\/users\/fludee01\/TEST_PGMS\/checkpoint\/serial\/serialcount 0 myout.txt<\/p>\n<\/blockquote>\n
<\/p>\n
Checkpointing in the absence of the -k option<\/h2>\n
If a job has been submitted without the -k<\/b> option and there is a need to checkpoint it, it can be done manually.<\/p>\n
The first thing to do is to find out the job number and the execution host. This is done with the bjobs<\/b> command:<\/p>\n
![\"\"](\"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-content\/uploads\/sites\/342\/2020\/03\/bjobsmarked.png\")
<\/p>\n
Next ssh to the execution host. Once there, issue the command:<\/p>\n
\npstree -a -u your_user_id -p<\/code><\/pre>\n<\/blockquote>\n![\"\"](\"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-content\/uploads\/sites\/342\/2020\/03\/pstreemarked.png\")
<\/p>\n
Look for the res<\/i> process. This is the LSF process that is reserving the resources on the node for your job. There may be more than one if you have several jobs running on the node.Identify the one you want by looking at the first child process and noting the last part of the command name. This should be the job number you are looking for (yellow highlight). This is your job file.<\/p>\n
The next child process is the LSF job script. It, too, has the jobid<\/em> as the last field in the file name.<\/p>\nThe next child should be the one you are interested in. It is the process started by the job script that is running your calculation. Note the process ID (peach highlighted). In this case, it is 331686 and happens to be the first process in a parallel job that has 3 other processes.<\/p>\n
Issue the command:<\/p>\n
\ncriu dump -t processid -D checkpointFolder --shell-job \r\nE.g.: criu dump -t 331686 -D chkpnt --shell-job<\/code><\/pre>\n<\/blockquote>\n <\/p>\n
Restarting Manual Checkpoint<\/h2>\n
To restart such a checkpoint, issue:<\/p>\n
criu restore -D checkpointFolder --script-job<\/code><\/pre>\nwithin a LSF job script. e.g.:<\/p>\n
\n#BSUB -q test
#BSUB -P acc_hpcstaff
#BSUB -W 50
#BSUB -n 1
#BSUB -R affinity[core(3)]
#BSUB -oo %J.out
criu restore chkpnt –script-job<\/p>\n<\/blockquote>\n
Note that in this case, the process being restored is the parent process of 3 parallel processes. Hence, the LSF options must make a request for 3 cores. If the checkpointed process is only a single thread, you need only request 1 core.<\/p>\n
[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"
Scientific Computing and Data \/ High Performance Computing \/ Documentation \/ Restart Your LSF Jobs: Job CheckpointCheckpoint Restart Checkpoint\/Restore In Userspace, or CRIU (pronounced kree-oo, IPA: \/kr\u026a\u028a\/, Russian: \u043a\u0440\u0438\u0443), is a Linux software application. It can freeze a running container (or an individual application) and checkpoint its state to disk. The data saved can be […]<\/p>\n","protected":false},"author":600,"featured_media":0,"parent":35,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_et_pb_use_builder":"on","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"class_list":["post-5859","page","type-page","status-publish","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/5859","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/users\/600"}],"replies":[{"embeddable":true,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/comments?post=5859"}],"version-history":[{"count":8,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/5859\/revisions"}],"predecessor-version":[{"id":7948,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/5859\/revisions\/7948"}],"up":[{"embeddable":true,"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/pages\/35"}],"wp:attachment":[{"href":"https:\/\/labs.icahn.mssm.edu\/minervalab\/wp-json\/wp\/v2\/media?parent=5859"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}