When to cut it?

Hi everyone, my chilies have been growing a ton but its getting colder here day by day (low 4c/39f highest 20c/68f) it this a good time to remove them from my garden, they still have peppers on or should i wait more?