Please not everyone that this blogpage is still under construction. I posted it so long so that those of you who only need a rough guide, could start so long. Let me know of errors you may spot, and I will gladly correct them. I will try and correct this entry asap!
The following post wil explain in some detail how to run a group analsyis on the CHPC cluster. Please read throught the UNIX tutorials referenced in this blog, before continuing. You will need a basic grasp of the command line. Your files also need to be raw dicom files and us need to be using t1_mprages. Check this on your scan protocol.
Step 1:
Login with (Remember to insert your own user name!):
yourusername@login02: ssh yourusername@sun.chpc.ac.za
Now change to the cubic directory…
yourusername@login02: cd /SCRATCH4/cubic
If you get an error message, that says you do not have permission to change to this directory, please let us know! CHPC then still needs to add you to the cubic group.
Step 2: Environment Variables
Next you must setup your “enviroment variables.” To do this, follow this link: How to Install Freesurfer
Step 3: Install our tools for running freesurfer on the cluster.
Run this command, while in the /SCRATCH4/cubic directory (Remember the dot!):
yourusername@login02: . clustersurf.sh
Remember this command, as whenever we need to update our scripts, you will have to rerun this command from the cubic directory!
Step 4: Copy your files onto the cluster
This step can be potentially done from anywhere. If it seems to slow, you will have to upload your files directly to the cluster.
Start by changing into the subjects directory:
yourusername@login02: cd $SUBJECTS_DIR
Check that you are in the right directory:
yourusername@login02: pwd
Your source files you should get directly from the protea server. To do this, follow this link:
When you have saved all your files into a directory, make sure the directory structure is correct. There should be one main directory containing all your files. Inside this directory should be individual subject files. In each individual files should be a few hundred .IMA files. Navigate using the basic UNIX commands.
Next you have to use the command scp…
Change to the directory that contains your main folder with all your subjects in it. The folder that contains all your subjects, we will refer to as your data directory from now on.
You use scp in the following way:
yourusername@login02: scp [data directory or source directory] [target directory]
When copying to the cluser, as it is through a network, you need to give its whole network address: sun.chpc.ac.za. You will also have to tell scp where your freesurfer subjects directory is. To find this, you need to login to CHPC, and run this command:
yourusername@login02: echo $SUBJECTS_DIR
Then you give all this info to scp:
yourusername@login02: scp -vr yourusername@sun.chpc.ac.za:/yoursubjectsdir/yourfiles
Note the -vr flag is for verbose, ie. it prints out all the information, and r is for recursive, that is scp copies all the files in your data dir.
So for instance if your data directory is myfiles_data, located in the /Users/bill/ directory, and your username is bmurray, it will look something like this:
bmurray@login02: scp -vr /Users/bill/myfiles_data bmurray@sun.chpc.ac.za:/SCRATCH4/cubic/bmurray/freesurfer/subjects/
Note: Remember the “/” at the end and before SCRATCH4!
Next, you need to create your target directory. This is where freesurfer will store the finished files. Say you want to call it myfiles_target, you would run the following:
yourusername@login02:mkdir $SUBJECTS_DIR/myfiles_target
Tip: If you get stuck with scp, try googling scp, and figure out what you are doing wrong. You can also try the alternative GUI option:
Also: Check in your subjects directory, when you start to copy, wether the files are correctly copying to the subjects dir. Should you discover you are copying the wrong files, press ctr-C to stop. Simply:
yourusername@login02: cd $SUBJECTS_DIR
Remember that should you leave the CHPC terminal open for few minutes without doing anything, it will lock, or freeze. Simply close the terminal and reconnect.
Step 5: Unpack your dicom files your files onto the cluster
yourusername@login02: cd $SUBJECTS_DIR
yourusername@login02:dcm_unpack2.sh
Tip: Use tab to complete your commands!
If this does not work, cd ~/clustersurf_bin and check if the script files are there. Should they not be there, try re-running the clustersurf.sh from the cubice directory. If they are there, run the command directly from that directory as follows:
yourusername@login02: cd $SUBJECTS_DIR
yourusername@login02: . ~/clustersurf_bin/dcm_unpack2.sh
The following should appear:
Please enter your data directory:
apathy confl_2 confl_9 confl_data_group_4.zip
APATHY confl_3 confl_data_group_1 confl_data_group_5
If your data dir is “confl_data_group_1”, just type it in. Tab wont work here unfortunately.
You should see the following output:
9440
Subject Number 1: ADAMS_RONNIE is submitted to Moab for Dicom unpacking.
Single Subjects dir is /SCRATCH4/cubic/splessis/freesurfer/subjects/EONCS_TEST_DATA/ADAMS_RONNIE/
… for each subject submitted.
Next, check that it is running on the cluster by typing:
yourusername@login02:showq | grep
You will see a list like this:
6717 jfouche Running 8 1:04:24:24 Thu Aug 18 01:06:15
6695 jfouche Running 8 1:19:48:30 Thu Aug 18 16:30:21
6692 jfouche Running 8 1:17:52:21 Thu Aug 18 14:34:12
6708 jfouche Running 8 1:20:28:08 Thu Aug 18 17:09:59
6715 jfouche Running 8 1:06:51:56 Thu Aug 18 03:33:47
6685 jfouche Running 8 1:08:20:32 Thu Aug 18 05:02:23
6745 jfouche Running 8 15:57:20 Wed Aug 17 12:39:11
6729 jfouche Running 8 1:06:50:27 Thu Aug 18 03:32:18
6687 jfouche Running 8 1:15:15:21 Thu Aug 18 11:57:12
6728 jfouche Running 8 1:07:06:53 Thu Aug 18 03:48:44
6712 jfouche Running 8 1:07:42:34 Thu Aug 18 04:24:25
6696 jfouche Running 8 1:20:23:30 Thu Aug 18 17:05:21
6686 jfouche Running 8 1:12:15:43 Thu Aug 18 08:57:34
6711 jfouche Running 8 1:08:19:28 Thu Aug 18 05:01:19
6713 jfouche Running 8 1:07:35:06 Thu Aug 18 04:16:57
6714 jfouche Running 8 1:07:31:54 Thu Aug 18 04:13:45
8682 jfouche Idle 8 1:21:00:00 Wed Aug 17 08:55:00
8684 jfouche Idle 8 1:21:00:00 Wed Aug 17 08:56:07
8686 jfouche Idle 8 1:21:00:00 Wed Aug 17 08:56:07
8687 jfouche Idle 8 1:21:00:00 Wed Aug 17 08:56:07
“Idle” means you need to still wait. “Blocked” means your Idle queue is full. Check again in about half an hour. This process should be quick. Else check periodically till your queue is empty.
To exit the cluster at any time, type “exit”. Don’t worry! This wont cancel your jobs!
To cancel your jobs, run the following command. (Note the capitals.)
yourusername@login02: canceljob ALL
Step 6: Run listmaker.sh
Next, change again to your SUBJECTS_DIR and run the next command:
yourusername@login02:listmaker.sh
The data dir is your data directory and the subjects dir is the directory which you made to put all the files in. Our example was myfiles_target, if you recall.
To check that listmaker did its job, change to your data directory and run:
yourusername@login02:cat t1_mprages.data
If the file t1_mprages.data does not exist in your data directory, check if dcm_unpack2.sh might not be still running on the clsuter. Try re-running everything again, and check if you have made any mistakes.
Tip: For more info on moab, follow this link :
Step 7: Start the recon process
Now you are finally ready to recon all your files! This process can take up to a week for a group of a hundred subjects.
yourusername@login02:cd $SUBJECTS_DIR
yourusername@login02:Recon_all_DICOMS_moab.sh
Enter your subjects dir and target dir as before.
You should see the following output:
9572
WoRkiNg…
Cleaning /SCRATCH4/cubic/splessis/freesurfer/subjects/EONCS_TEST/EXAMPLESUBJECTNAME.2.2008.09/
Subject 44 of 52: EXAMPLESUBJECTNAME.2.2008.09 submitted to cluster
Again use showq, as before to check on the progress of your recon.
Step 8: Checking on your files:
Change to your target directory. You should see a lit of subject names and OUTPUT files. The OUTPUT files are plain text files, which contains info on which subjects it submitted and to where. Use cat to access them, like you did with the mprages.data file.
Now try the following commands, running them while you are in your target directory, tabbing to complete the subject names:
yourusername@login02:ls EXAMPLE_SUBJECT_20090703_001/scripts/recon-all.done
yourusername@login02:ls */scripts/recon-all.done
yourusername@login02:ls */scripts/recon-all.error
yourusername@login02:ls */scripts/IsR*
You should see which files have finished, which have errors and which are still running.
Use …
yourusername@login02:cat EXAMPLE_SUBJECT_20090703_001/scripts/recon-all.error
… to get the error messages.
Step 9: Copy back to your own PC for error checking:
This time, you will use the same scp command, but in reverse:
scp -vr bmurray@sun.chpc.ac.za:/SCRATCH4/cubic/bmurray/freesurfer/subjects/myfiles_target /Users/bill/
Remember that the output files are far bigger than the input files, so you might have to report at CHPC directly…
The CUBE 