I can understand that being called a midget could be received badly mainly because it doesn’t sound really good. I had a fatass bully in high school who called me a midget even though we were the same height, but that’s a different story. Dwarf or dwarfism I don’t understand to be a bad word, especially since it’s used in medical terms as Gina said. If using dwarf was bad, then you couldn’t use it for characters as their bodytype (although they use it as a race, but it’s not technically a race). Couldn’t use it for Gimli, Bronzebeard, level 3 Crotchcrusher with a +2 attack mace, etc.
The word usage of dwarf is extremely confusing. You can’t use it to describe a person apparently, but it’s just dandy in fantasy movies and games for some reason. Doesn’t make sense.
But I think as far as medical terms I think midget is just that you are proportionally smaller in size, but dwarfism your bone structure is hindered, such as misgrowing femurs, bad hips, pronounced forehead, etc. I could be wrong but that’s what this swine honkey cracker gathers.