What are these bugs in my Serrano plant? Are they harmful?

We recently had some heavy rain in California, I woke up this morning to see this tiny worm infestation in the soil of my pepper plant :/
I feel like I might need to repot the whole plant now because there’s so many and there no telling how many more are moving thru the soil.

what should I do about this? Should I scoop them out or leave them?