The yellowing is totally normal at this stage...and will progress as "fall colors" start in. Go ahead and remove any yellow or dead leaves, as these can cause issues with mold/pests. As for removing the fans...I would say you can get away with removing a few, but I wouldn't go crazy. I'm not really an outdoor grower, but I know most folks that do kind of limit how much defoliation they do, because the plant needs those leaves. Lets see what others say as they chime in.
@Weed Warrior got any advice?
Hope it helps