Yep, remove exhaust. As well, in order to drop the exhaust down far enough, you have to either remove the transmission brace, or take the exhaust loose at the muffler(or converter) I believe. In my instance, I brought it in to work(there's a garage and full time mechanic here) and had him do it. Instead of removing the entire exhaust or brace or anything, we took the exhaust loose on the left side only, from manifold, cut the pipe and removed that section, put the new starter on, then hooked back up to the manifold and welded the pipe back together. Not totally kosher I suppose, but vehicles here in my state are only inspected when bought/sold... so... :)
Good luck!