I think youre right! They move by scrunchin up and then inchin along. And ive not seen any yet on the soil at all, just on the leaves at the moment
Yup, inchworms, loopers, whatever you call them where you are. I don't want to pinpoint your location, but if you are Southern US could be California Spanworm? SO many kinds of loopers.. See if you can spot a moth visiting your ladies.
Diatomaceous earth won't get those guys, but it will get a lot of other pests so no harm in using it. They lay eggs under your leaves, hatch, eat away as they go through a few instar stages then you get a moth. The moth lays eggs, on and on. They probably have hot moth sex on your plants too

Not much you can do if you are outdoors for the moths, but a regular application of Bt will kill off what hatches as they eat your plant (problem is they DO have to eat some of your plant, but won't be as bad as untreated), as the moths will keep laying eggs.
I believe it is safe to use right up till harvest on vegetables, but as for a smokable product hmmmmm
@Waira what's your take on Bt in flower?