Initially, I thought it might be a crack in the header like you're talking. But I tried putting a hand in and feeling for leaks. I've done that on my old CJ a few times and found leaks every time. But I didn't feel anything on the XJ. I can usually hear where the leak is too, but no luck with that on the XJ either.
I was actually hoping it would be a cat problem that was causing the joint between the header and exhaust pipe to open slightly. But really, since it goes away after a few minutes, it's probably not a cat problem and is more likely a small crack somewhere that I just can't feel. A cat would be easier to fix. If it's a manifold leak, I'll just wait till it starts spewing fumes before I touch it.
SWMBO probably wouldn't let me hook the vac up to the tailpipe - might get it dirty ya know. Otherwise, I'd give that a try.
Ya know what else is odd, but it came like that from the factory, is that the exhaust pipe has a serious crimp in it right where it makes the turn leaving the engine bay. It's crimped to about half diameter. My cousin has an XJ that's the same way. Very strange.