I have found that it is usually safe to ballast everywhere except around the moving parts of a turnout; i.e., between the moving point rails and at both ends of the throwbar.
I paint the roadbed gray before installing a turnout so the areas that are not covered with ballast are not as obvious, since the ballast is also gray.
I have walked real railroad tracks and observedd that they don't ballast around the moving parts of turnouts, either. There is some ballast, but not up to the tops of the crossties like it is elsewhere along the line.