the only stuff you have to worry about scraping off is the stuff you *can* scrape off. if it's really stuck on there, there's no harm in leaving it up.

using a solvent to remove a coating from a porous surface not a good idea. nor is sanding old paint that may contain lead. just scrape off the loose stuff and wear a mask. then a coat or two of primer and you're good to go.