I've long been using clumping litter (the kind you're being encouraged to avoid) for another purpose - as a component of my Cell-U-Clay-based ground goop. It imparts a limestone color and texture, allowing me to craft rock faces "carved" in place. Actually, it's more like sculpting in place; I use a plastic spoon to both hold the material and to trowel it on, working in strata lines with the edge. The conbination of Cell-U-Clay's lumpiness and the litter's grittiness and tendency to crack results in good-looking stone surfaces with relatively little effort. Of course, one can fine tune later on just by carving when dry, or by re-wetting and working it additionally; this must be done with large cracks (that do develop) to close them up, but the reworking just makes it look better still.
As a ballast material, one might consider using clumping litter as an admixture to other non-dissolving ballast, since the clumping litter will soak in the glue and will itself act as a glue for the rest. Mixing litter with other materials will dilute the smell, and give you better control over color & texture, while extending the amount of coverage you get out of the more expensive ballast.