I think it is an accepted knowledge that the word 'than' should be thought of as a conjunction in a comparatve sentence if you want to write in a formal setting, but what I have difficulty with is knowing when what follows the subject form of a word that comes after the 'as' is optional and when not. Please help.

John is taller than I (am) -- the word in parentheses seems to be optional.

John plays better than I (can play) -- the words in parentheses seem to be optional.

John dances better than I (can dance) -- Again, what is in parentheses seem to be optional and to me the sentence 'John dances better than I do' sounds good too.

... so far so good, if my previous effort was somewhat fruitful, but what about this?

John plays chess better than I do -- Here, 'can play' seems awkward to be used, so I used the word 'do'. Is using 'do' correct? Also, not placing 'do' at the end seems to be incorrect. Is placing 'do' at the end a must?

Can you think of any situation in which the placing of some word after a subject form after the word (conjunction??) is not optional?

Sorry, if my post was long.
You can use the full form, the 'pro' form, or just the pronoun. In the last case, note that isolated subject pronouns after than are often expressed as object pronouns in informal situations. The 'pro' form is the echo of the operator (modal or auxiliary) -- do if the first clause has no other operator.
Joe can play chess better than [I can play chess / I can / I / me].
Ed will reach the train station sooner than [we will reach the train station / we will reach it / we will / we / us].
Karen saw the sign more clearly than [he saw the sign / he saw it / he did / he / him].
It's very common in sentences of this sort to drop everything after the pronoun. So "John is taller than I" is the acceptable wording. In fact, in conversation many, probably most, people would say, "John is taller than me." To those folks using the "correct" pronoun seems pretentious. So your final example would read "John plays chess better than I." Mind you, it's not incorrect to add "do" or to repeat the verb "can dance," or"can play," but most native speakers would not do that, and it would sound odd to many. If you wanted to sound like a native speaker, however, you would use the object form even though your text would tell you it's incorrect.
 CalifJim's reply was promoted to an answer.