3 sentences with the same meaning?

All three are grammatically correct.
"I'm interesting in this book" is not a possible rewrite, however.

They are all equally acceptable in both formal and informal contexts.

Each has its own slightly different meaning.

The third is the most objective because there is no mention of "me". It asserts that the quality of being interesting (adjective) applies to "this book". Even so, it's just the opinion of the speaker, of course. "You should read this book. It's interesting. Well, anyway, I found it interesting. I think you will find it interesting as well."

The first sentence might be heard in a book store. The saleslady approaches you. "Can I help you?" she asks. You say, " I'm interested in this book". This comment can be used to mean, "I am interested in buying this book".

"This book interests me" could also be used in the book store example, but it's not as strong in terms of suggesting that you would like to buy it. This sentence might also be used in a case where the physical qualities of the book are highlighted. "This book interests me because of the unusual binding. I think it was probably made in the eighteenth century. That's why it interests me."

I'm sure others will be able to contextualize these sentences in other ways.

CJ