relative clause

C: countable nouns that are used with a or an or a number and have a plural: car, soldier

U: uncountable nouns that cannot be used with a or an or a number and have no plural: happiness, pasta

(On the inside front cover of Macmillan Essential Dictionary for Learners of American English)

I think the above grammar codes should be rewritten as the following:

C: countable nouns, which are used with a or an or a number and have a plural: car, soldier

U: uncountable nouns, which cannot be used with a or an or a number and have no plural: happiness, pasta

Am I right?

I think the above grammar codes should be rewritten as the following: ...... Am I right?

Yes.  Given the context you describe, I certainly would think so.
When the relative clause is essentially a definition of the antecedent (or nearly so), it doesn't seem to make much sense to use the restrictive relative pronoun that.

Here are some equally silly examples in the same mold:

Elephants that are bigger than mice are said to be fearful of small animals that squeak.
Identical twins that are siblings with identical genetic material usually look almost exactly alike.
The sun that is a rather ordinary star is the center of our solar system.

CJ