I agree. Mold spores are everywhere. They're everywhere already. You control it by regulating the moisture. Tobacco molds less easily than bread, and maybe about as easily as cheddar cheese. Tobacco that is stored dry but pliable usually won't mold.
Stems are more hygroscopic than lamina, so they tend to mold more easily. I never worry about using leaf with a moldy stem. I just rip out the icky stem, and use the leaf. Scant mold on the leaf is, like MarcL has stated, inconsequential. If a stemmed leaf smells moldy, I always toss it.
Since everybody's storage and usage environment is different, you just have to experiment. My usual goal is to store leaf or leaf scraps as dry as possible while still being flexible.
Bob