I didn't see you say what scfm rating your exhaust fan has? How about the carbon filter rating? Passive intake or do you have an intake fan as well?
As a HVAC service tech I'll chime in.
@Bud Wiser UK and
@BigSm0 have already said treat the intake air and they are spot on! It's as good as treating the cab air. A dehumidifier in your bedroom will do the trick. I'd hate to see you loose bud to the rot!