Did my first paid-ish detail and now I'm second guessing everything

On knowing when you're done, work top to bottom and left to right, one zone at a time, and don't circle back. If a door pocket gets one full pass of clear out, wipe, dry wipe, it's done, move on. Going back over the same spots is how you burn an hour without changing anything. A decent trick for judging yourself is opening the door and looking at the sill and jamb area in daylight, that's where missed dust shows up first. If the sun-facing surfaces look clean from two feet away, the interior is done. A first paid job at $40 doesn't need to be perfect, it needs to be noticeably better than before, and it sounds like it was.

The dusty film that keeps coming back might be oils from hands mixed with dust, not just loose dust. When I did my brother's van I kept wiping the steering column and dash with a wet cloth and it always looked hazy again. What finally worked was spraying a bit of the cleaner right on the cloth, scrubbing the textured parts with a soft brush, then wiping with a second clean damp cloth to pick up what the brush loosened. Two cloths, not one.

Do you let the cleaner sit on the plastic for a minute before wiping, or wipe it off right away? I couldn't tell if I was rinsing it off too fast.

I'd push back on the 'one pass and move on, never circle back' advice for someone on thier first paid job. That works once you've got a feel for how dirty a panel acutally is, but early on you don't know what clean looks like yet, so you finish zones that were never really finished. When I started doing friends' cars I made the opposite mistake and it taught me more than the zone method would have. What I'd tell OP is circle back once, at the very end, with fresh eyes and good light, then stop. Also those dried mats probably needed a stiffer brush worked in before rinsing, not more shampoo.

Agree with the guy who said that haze on textured plastic is skin oils mixed in with the dust, that was my exact experience. What made the difference for me was letting the diluted cleaner sit on the trim for a minute before wiping, then using a cheap paint brush from the hardware store on the grain, wiping with the grain not in circles. The dried-out mat thing is normal too, they rarely look new again, they just look clean tired instead of dirty tired.

On the mats, one thing that helped me was letting them dry completley, then hitting them again with the vacuum using the brush attachment. The fibers sort of mat down while they're wet and look flat even when they're actually clean, but a good once-over after they're dry fluffs them back up. I also stopped judging mats while they're still damp, they always look rough at that stage. Next job try taking them out and doing them in the driveway, way easier to see what you're actually getting out of them than working inside the car.