The room was clean when I went there yesterday because someone had cleaned it four days ago.
"to come" works better when the action is toward the area the speaker is in. It should work in the same way that 来る does.
It's also uncommon to use "three days before yesterday", it would be more natural to say "four days ago." After "the day before yesterday" it's much more common to say the number of days, and not using yesterday.