gritty? sounds like you're putting it on too thin.
sounds like you haven't controlled the suction enough either. Most people put two coats of pva on. First coat 1:5, let it dry thoroughly. Then stick another coat of pva on, this time at 1:3. Skim it when it's gone tacky.
what type of walls are you trying to reskim? Were they done with gypsum based plaster or lime?