Yes, I think, otherwise, we have to believe that the subjects of math and physics are intrinsically difficult. But for some people learning music would be more difficult. So 'difficulty' is a subjective thing.
On the other hand, the more accomplished mathemeticians and physicists might claim that their subject is as 'soft' as 'the creative arts,' that math and physics are as speculative and conjectural as any other category of knowledge.