Isn't a rule/are no rules

There is virtually no difference.  The context may influence your choice, however.
For example, if you are seeking a single rule, you may want to report your negative findings as There isn't a rule to ...  But if you are seeking several rules, you are more likely to report the result of your search as There are no rules to ...

There is nothing to prevent you from choosing either expression for either result, however. 

And there are also There isn't any rule to ... and There aren't any rules to ..., just in case you'd like even more choices! 

CJ

No, those are not correct. I'm sure we can think of the odd exception if we try hard enough, but generally the rule is that the verb must match the noun, 99% of the time.