In addition to the points raised by others, it would also be useful to distinguish between historical/cultural eras where objections to polygamy differ.
For example, in the United States the Protestant Christian tradition dominated responses to polygamy as "immoral" because most traditions only recognized sex with one marriage partner as morally allowed by God. Thus the Mormons were attacked not on that basis primarily, since contemporary Christian society viewed multiple wives as simply open, socially sanctioned promiscuity. (Hypocrisy aside, because of course men cheated - they just didn't expect social sanction for extramarital affairs.) They may have argued that it was bad for the wives, too, but the basis for that badness came from the sexual argument - that men with more marriage partners were using the women for sex. In a monogamous relationship, the "partnership" aspects of the marriage seemed more obvious, and why else would a man want more wives except for sex? Thus degrading them all.
Conservatives today still think that polygamy is wrong because of the sex argument. Now, however, we've also got feminism, which objects to polygamy on the grounds that it is bad for women because women are used as reproductive units subservient to their husbands, unable to have lives outside of the narrow category of "motherhood." The validity of this critique changes depending upon the culture to which it is applied: in the United States, where women are already seen as more than just their reproductive systems, the critique is valid. But as applied to other cultures, I would hesitate to give a blanket statement that polygamy is wrong. Mostly this is because gender relations elsewhere don't come from the Victorian Christian tradition (which American feminism is responding to) and so our assumptions about what women's roles are/should be might not describe what is actually happening. (This doesn't mean that gender discrimination is okay elsewhere, just that we should be informed of reality and our own biases before becoming the arbiters of True Morality And Justice.)
Note that both conservatives and feminists say that polygamy is bad for women, but their underlying rationales are different. Polyamory, then, might be okay to feminists, because it is supposedly based upon mutual love and respect for multiple partners rather than upon assuming women's "purpose" is to reproduce for men.
Polygamy as a natural comparison with gay marriage only makes sense from a "society is now endorsing forbidden sex" point of view. But for liberals/feminists, polygamy and gay marriage are not comparable. Polyamory and gay marriage may be, however.