Principles of spatial processing
Try [http://www.fil.ion.ucl.ac.uk/spm/doc/books/hbf2/pdfs/Ch2.pdf John Ashburner's realignment chapter] in the [http://www.fil.ion.ucl.ac.uk/spm/doc/books/hbf2/ Human Brain Function book] - second edition.
See also the other chapters on "Computational anatomy" in [http://www.fil.ion.ucl.ac.uk/spm/doc/books/hbf2/ HBF2].
Mark Jenkinson and Steve Smith have an excellent introduction to the problems of optimizing in registration - see the [http://www.fmrib.ox.ac.uk/analysis/techrep/tr00mj2/tr00mj2/index.html HTML version] or [http://www.fmrib.ox.ac.uk/analysis/techrep/tr00mj2/tr00mj2.pdf pdf version].
You may get some benefit from the theory pages at the [http://bishopw.loni.ucla.edu/AIR5/index.html AIR website]. AIR is a realignment / coregistration / normalization program written by Roger Woods of UCLA. His pages go into details of the implementation of spatial transforms, some of which are relevant to SPM.