With a smoke test you pump smoke into the booster and if there are leaks it will leak out around the booster. A smoke test is a reliable way for locating vacuum leaks.
There is a one way valve that is used to hold vacuum as well, that could be damaged, but it wouldn't explain why the issue is less noticeable when the car is driving. Vacuum is highest when the throttle plate is the most closed.
Foot off the gas = throttle closed = high vacuum
Pedal to the floor = throttle all the way open = low vacuum.