Yes, it's common in the U.S. to say "it's ten of five", meaning "ten minutes before five o'clock" -- but don't worry about why we say it that way, or the meaning of the "of" -- you'll only get a headache! Just accept it as an idiom.
(I've been saying it all my life, and now, suddenly, it seems ridiculously strange to me. It almost sounds as if it means "ten, selected from a group of five" -- it reminds me of the Star Trek character called "Seven of Nine.")