For trans men, phalloplasty is rare because it is a very invasive, extensive, and expensive procedure. Not as much for a vaginoplasy for trans women, and those are not nearly as rare.
Even if all bottom surgeries were rare, though, it wouldn't change the fact that saying you are not attracted to trans people because of their anatomy is a blanket statement that depends on the speculation of what genitals they have.
Since we know that cis people often share physical characteristics with trans people--cis women with facial hair, high foreheads, inset eyes, even Adam's apples; cis men who are short, have high voices, maybe even wider than average hips--the only way to determine definitively if someone were trans (pre genital reconstruction) would be genitals. The only way after genital reconstruction would be something invasive such as karyotype testing or self disclosure from the trans person.
Since the main difference between a trans person and a cis person is the circumstances of their birth, a blanket statement that you don't like or won't date trans people is by definition transphobic.