There are some rules to help you remember. This chart always helps my students.
ir - used before r - irrelevant, irresponsible
il - used before l - illicit, illogical, illegal
im - used before p,b,m - imperfect, impossible, immoral,
in - used before all other letters - uncomfortable, unexceptable
un, non, and dis have no rules and require a little memorization