The "had" part is correct.
You can try "even if he hadn't boxed" or "even if he hadn't been a boxer." I prefer both of those to "hadn't done boxing."
However, there are a couple other problems:
We all know the example of Muhammad Ali. However, we don't know if his illness is connected with all the blows he received to his [the is okay too] head, or maybe he just had bad luck, and even if he hadn't been a boxer, he would have suffered from the same illness anyway.