fMRIPrep at MRC CBU

For details about what fMRIPrep is and why you'd want to use it, see their website. This page is about how you use it at CBU with Singularity. Notice that fMRIPrep has its own documentation on Singularity use, so please consult that also.

What's Singularity?

Singularity is a container image runner, like the more famous Docker. The preferred route for using fMRIPrep is as a Docker container image, which we run in Singularity rather than Docker. Easy right? There is a good explanation on this CBU Intranet page.

Wait, what's a container image?

You can think of it like a little virtual machine that bundles a version of Linux, all the dependencies fMRIPrep needs, and the fMRIPrep application itself into a neat little package that can run independently of the rest of the imaging system.

Using fMRIPrep at CBU

We keep containers for each release at these imaging system paths:

/imaging/local/software/singularity_images/fmriprep/
├── fmriprep-1.1.8.simg
├── fmriprep-1.4.0.simg
├── fmriprep-1.4.1rc4.simg
├── fmriprep-1.4.1.simg
├── fmriprep-1.5.0.simg
├── fmriprep-20.2.1.simg
├── fmriprep-20.2.6.simg
├── fmriprep-21.0.1.simg
├── fmriprep-22.0.0.sif
├── fmriprep-22.1.1.sif
└── fmriprep-23.1.4.sif

To use it, you would do something like this frankly intimidating command, substituting the version that you want to use:

singularity run -C -B /imaging/jc01/kamitani:/kamitani /imaging/local/software/singularity_images/fmriprep/fmriprep-1.1.8.simg /kamitani/bids /kamitani/fmriprepnew participant --participant-label sub-03 -w /kamitani/fmriprepwork --nthreads 16 --omp-nthreads 16 --fs-license-file /kamitani/license.txt --output-space T1w

It's a bit less intimidating if you break it over multiple lines (but harder to copy and paste successfully):

   1 singularity run -C \
   2     -B /imaging/jc01/kamitani:/kamitani \
   3     /imaging/local/software/singularity_images/fmriprep/fmriprep-1.1.8.simg \
   4     /kamitani/bids /kamitani/fmriprep participant \
   5     --participant-label sub-03 -w /kamitani/fmriprepwork --nthreads 16 \
   6     --omp-nthreads 16 --fs-license-file /kamitani/license.txt --output-space T1w

Again, computing has a nice explanation of what it all means on this CBU Intranet page. The key things to notice are

Worked example: Handling custom templates, submitting to SLURM cluster

(thanks Joff Jones)

fMRIPrep supports custom templates, but getting them into your container can be challenging! There is information on this issue in the fMRIPrep docs. Below is a pattern that Joff Jones has found to work at CBU. The tricky bit here is keeping track of what happens natively (on the login node), and what happens inside the container.

Another tricky thing is to run the fMRIPrep job on the cluster. It is bad practice to run these fairly intense jobs on the login nodes. The example below takes you through one way to do this. You could run this in an iPython shell session, or put the examples into a Jupyter Lab notebook.

First you need templateflow on your Python path, so try something like activating Neuroconda.

Then we want to start by downloading the template we want to use:

   1 import templateflow.api
   2 templateflow.api.TF_S3_ROOT = 'http://templateflow.s3.amazonaws.com'
   3 get('MNI152NLin6Asym')

After that, we can write a little Python shell script that submits the job.

   1 # setup template flow environment
   2 my_env = os.environ.copy()
   3 # tell templateflow where to look for the templates *inside the container*
   4 my_env["SINGULARITYENV_TEMPLATEFLOW_HOME"] = "/templateflow"
   5 
   6 # directories
   7 bids_dir = '/path/to/bids'
   8 slurm_output_dir = '/path/to/cluster/log/files'
   9 os.mkdir(slurm_output_dir)
  10 
  11 # set range of BIDS_ID numbers to be processed
  12 for n in range(1, 100):
  13     p = subprocess.run(" ".join(["sbatch", "--mincpus=6", "--time=48:00:00", #sbatch cmd
  14                          "--job-name=fmriprep",
  15                          "--output", slurm_output_dir + "/sub-%03d.out" %(n),
  16                          "singularity", "exec", "-C", #singularity call
  17                          "-B", "/imaging/jj02/CALM:/CALM", # freesurfer license location
  18                          "-B", bids_dir + ":/bids", # bids directory
  19                          "-B", "/home/jj02/.cache/templateflow:/templateflow", # these might need to be set to your home
  20                          "-B", "/home/jj02/.cache/fmriprep:/home/fmriprep", # these might need to be set to your home
  21                          "-B", "/tmp:/tmp",
  22                          "/imaging/local/software/singularity_images/fmriprep/fmriprep-1.5.0.simg", # singularity image
  23                          "fmriprep", # run fmriprep
  24                          "/bids", # bids directory
  25                          "/bids/derivatives/fmriprep-1.5.0", # fmriprep directory
  26                          "participant", "--participant_label", '%03d' %(n), # participant info
  27                          "-v", "-w", "/bids/derivatives/fmriprepwork-1.5.0", # wording directory
  28                          "--skull-strip-template", "MNIPediatricAsym:cohort-1", # child skull-strip template
  29                          "--output-spaces", "MNIPediatricAsym:cohort-1:res-2", "T1w", # child template
  30                          "MNI152NLin6Asym:res-2", "MNI152NLin2009cAsym", # For ICA-AROMA & carpet plot
  31                          "fsaverage", # For freesurfer BBR and surface-based BOLD
  32                          "--use-aroma", # ICA-AROMA denoising output
  33                          "--fs-license-file", "/CALM/license.txt", # freesurfer license
  34                          "--write-graph",
  35                          "--fd-spike-threshold", "0.5", "--dvars-spike-threshold", "0.5",
  36                          "--notrack", "--resource-monitor",
  37                          "--skip-bids-validation"]), # skip this for cluster jobs as it tries to go online
  38                         shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)
  39     print(p.args)
  40     print(p.stdout.decode())
  41     print(p.stderr.decode())

Notice that several bind-mounts are used (the -B flags) to mount various paths that are needed inside the container, including the /templateflow path where the template we downloaded natively becomes available inside the container.

An example notebook that also implements MRIQC is available for download here.