Sooner the better I guess, it's not a great tasting thing. For caterpillar this year, I used Bt-i, bacteria thuringiensis israeliensis. It's a bacteria that messes with them, will make them stop eating immediately and kills them in a few days. Just sprayed it on once and did the trick. Maybe less residue than neem, you might find some for next time. What other bugs you got there that you're fending off?