I'm afraid to say that I don't think that there is always a rule, Guest. You'll master the different pronounciations of "o" after spending some time on pronounciation and listening. However, you can get some hint from your dictionary which gives the IPA (International Phonetic Alphabet) phonetic symbols to guide you how to pronounce the word (the British dictionaries, such as Oxford, Longman, ... However, you can use an American dictionary, they also provide you with some symbols which are not international and I don't know the pronounciation of them, but if you can, it's ok).  Then you'll see different symbols for "come/love/mother/above" which are pronounced as "but", "horse/force" before "r", ... The difference between these two sound is that in the latter case you give a rounded shape to your lips, whereas an unrounded position is taken when pronouncing the words of the first category. In "fond/fog/doll/rock", though stressed, the pronounciation is different (and also we have the American version and the British one). Words like "stone/focus/nose" again fall into another category with different British and American variations (and this sound is a diphtong, a combination of two vowel sounds). The rest are easier to recognise, since there is another letter ptresent to give you a hint, like "book/shoot/shoe", "boy/boil", "mouth/down,doubt/", however this doesn't always work: door, double, grow.

Look at your dictionary and check the pronounciation of the words and pay attention to them, meanwhile, I'll look to find some new information and rules, if there are any.

Good luck,

The on-line Cambridge Learners' Dictionary will show you the phoenetics of any word you look up, if you select "show phonetics".

http://dictionary.cambridge.org/

There is also a section on the phoenetic alphabet.

http://dictionary.cambridge.org/help/phonetics.htm