People tend to join the "winning" team as opposed to sticking it out with the "losing" team, as obviously people tend to enjoy winning more than losing.
Plus as mentioned above sometimes people get caught up with being the "best" and can't handle a single defeat, let alone a series of them.
Plus some people's ego just gets in the way of everyone else having fun, and drama takes over.
I love pvp and love all the players who do it, most of us are killing eachother for the same reasons etc, so I would love to see innovative ways of creating consistent and relatively even pvp (having more than double your numbers appear is pretty frustrating, unless it is a swarm of plebs).
Chopping people into 5 man teams (5 seems small though, 8 seems more fun and allows for more variation) doesn't stop those teams working together or abusing the system :\