Hello Sjlee,
No, I'm sorry, neither of those phrases are correct.
What's your name, what's his name, what's the name of that thing... those are all correct.
There are some expressions that are mild explatives - like What in the name of heaven is going on in here? - but what are you trying to say with the "what in name" one?