No idea what your battery is up to. That sounds like either your idle is very low, or the alternator regulator isn't quite hacking it. When the engine is running the alternator should be able to run everything, including headlights, without depending on the battery.
You should see 13.8V at the battery terminals ideally. It's worth having a multimeter to check this much. To be honest though I've seen multimeters cause more confusion than benefit amongst people who don't understand impedance -- for most stuff, stick with a test lamp!
Anything more needs a bit more spent on test gear and at least a serious read of a Haynes manual, otherwise throwing it all at an auto-sparky with a proper tester and 5 minutes. You can usually blag a freebie "ready for winter" test from somewhere, but be stern about them trying to sell you a new alternator immediately afterwards - they may not be the best price for it.