What's wrong with prejudicial attitudes such as heightism?
Well these attitudes often lead to discrimination.
When a short statured person is bullied, rejected by peers and romantic interests, denied job opportunities, etc. there is a problem.
Heightism affects people's lives in negative ways. That is reason enough to discuss and challenge it when it surfaces.