If you take the point of view that
perfect is an 'absolute' (non-gradable) adjective, an adjective of which there can be no degrees -- a point of view being disputed in this thread, but for the sake of argument, let's say we take this viewpoint -- then
more perfect is impossible because it expresses a degree of perfection, a quality which, by the original supposition, does not occur in degrees. Given that it is absolute, it is as pointless to talk about more or less perfection as to talk about more or less nothingness, or -- to pick a more famous example -- to talk about some pigs being
more equal than others.
more perfect, rather perfect, pretty perfect, a little perfect, fairly perfect, very perfect, and
somewhat perfect are all "banned" within this viewpoint because all express degrees of perfection.
Nevertheless, it is not contradictory, still remaining within the same point of view, to speak of conditions which are closer to or farther from the absolute (non-gradable) state of perfection. This is not at all the same as talking about different degrees of perfection. In this group we have nearly perfect, almost perfect, far from perfect, very nearly perfect, just about perfect, and pretty much perfect.
pretty is close in meaning to fairly. It belongs in the first ("banned") group because it expresses a degree of a (gradable) quality: pretty good, pretty small, pretty old, pretty shabby
pretty much is close in meaning to just about or almost. It has to be thought of as a two-word idiom not exactly derivable from the meanings of the two words that compose it. It belongs in the second group because it expresses a distance from an (absolute) quality: pretty much finished, pretty much empty, pretty much intact, pretty much equal
______________
If you take the point of view, on the other hand, that perfect is not an absolute adjective (i.e., is a gradable adjective), then all the adverbs of degree can apply: fairly perfect, very perfect, less perfect, etc. I've never, personally, taken this point of view. In fact, I was told in grade school that the "more perfect union" spoken of in the Constitution should really have been a "more nearly perfect union". Apparently that lesson has stuck with me. 
Yet common sense dictates that some expressions which take perfect to be gradable have become standard idioms in English and therefore must be accepted. But I don't think I am alone in saying that pretty perfect is not one of them.
CJ