Distemper is a term used to describe an old type of paint that was made out of powdered chalk. Once dry, it has a powdery surface so if you rub your hand over it you get a whitish/blue residue on your hands. PVA does not cure it, you need to seal it with a solution designed for the job. Or you can scrape it back.
If it's really bad then WBA/Bond-it will not stick to it either, unless you've either sealed it with the special solution, or scraped it back.
Im extreme cases it's often easier to take the wall right back to brick/stone and re-do from scratch rather than skim over the distemper - it can be a right pain in the a***!! (Obviously explaining the problem and passing all the cost onto the customer

)