That is very likely the thermostat. However, the heater core could be clogged up too.
The thermostat is the first step. Just buy the correct one from AutoZone, get some coolant (preferably 50-50 premixed). Drain coolant. Trace upper radiator hose to the engine. That is where the thermostat should be (not sure...haven't seen your engine). It should be in a housing with probably two bolts on it. Take the old one out (note the orientation..some have a jiggle valve). Put the new one, with new gasket, in and bolt it back down. Do not put it on really tight. If you do, you may crack the housing. If it isn't tight enough, it will leak a little after it gets hot, but you can tighten it while the engine is running until it stops leaking. You do not want to be looking for a new thermostat housing!
Then reconnect the hose, close the drain plug and fill with coolant. Run the car with the radiator cap off and the heater on full. Watch the fluid level and run it until it is hot. Add fluid as needed. If that doesn't fix it, it's probably the heater core.