It's always a good idea to remove the window glass first to avoid any uneccesary damage (scratches) or extra clean-up. However, generally the products used to strip paint safely off plastic bodies is safe for the window "glass" as well.
In most cases, the widow glass is not glued in and you just have to figure out how it was "snapped" in, they're kind of like chinese puzzle, ie: remove this piece before you can remove that piece. The use of toothpicks comes in handy to keep pieces in place until ready to be pulled out.