> I obviously don't want to break it, but am I just not pulling hard > enough?
Correct.
Whilst turning the slot around 45 degrees, use your other two hands to slide another scredriver down the volume knob side of the unit and lever it out with much more force than feels safe.