Well, reading your comments, my own, and looking it up in a dictionary, there seem to be two major uses for
by.
1)The first and most frequent use is that it specifies the means or manner in which something is accomplished. When you say the house was built by the men, you are not so much specifying the builder as specifying the means. A man lives by himself explains the manner in how he lives. Crossing the ocean by boat explains the means.
by taking short cuts
by boat
We go by the clock. (the reference to time is how they operate)
shorter by two inches - Explains how the shortness is accomplished, by two inches in this case.
2)The second use is near or passing by. walking past the house: walking by the house standing next to, standing by somebody (giving support by being close)
She lives on Jackson St. by 10 Ave. By- (This is not normally directions, this is location. On Jackson near or at 10th Ave.)
We will be ready by six o'clock. (near the time)
By the time we get to town, he will be ready. (Apparently specifying the closeness between when we get to town and when he will be ready.)
By the way, did you know that... ( Indicates a side comment, something not central to a discussion. Way is an old fashioned or not common word for road or path. Apparently an analogy between being at the side of a road (by the road) and something being less important)
So, what do you think?