I think here "let" is a modal auxiliary, and except for "ought to", they are followed by a bare infinitive. You'll find the same construction after "help", "make" (causative), and perhaps others, but I don't know the explanation for it, just that it's a grammar rule