Social selection is a mode of natural selection based on reproductive transactions and a two tiered approach to evolution and the development of social behavior. Reproductive transactions refer to a situation where one organism offers assistance to another in exchange for access to reproductive opportunity. The two tiers of the theory are behavioral and population genetic. The genetic aspect states that anisogamy arose to maximize contact rate between gametes. The behavioral aspect is concerned with cooperative game theory and the formation of social groups to maximize the production of offspring.
The theory was proposed by Joan Roughgarden as an alternative to sexual selection. In her critique against the neo-Darwinian defense of sexual selection, Roughgarden outlines exceptions to many of the assumptions that come with sexual selection. These exceptions include sexually monomorphic species, species which reverse standard sex roles, species with template multiplicity, species with transgender presentation, frequencies of homosexual mating, and the lack of correlation between sexually selected traits and deleterious mutation. Other researchers, such as biologist Mary Jane West-Eberhard and evolutionary medicine researcher Randolph M. Nesse, instead view sexual selection as a subcategory of social selection, with Nesse arguing further that altruism in humans held fitness advantages that enabled evolutionarily extraordinary cooperativeness and the human capability of creating culture.
Short for the genetic-portfolio balancing hypothesis, this idea, proposed by Roughgarden, is used as an alternative to the Red Queen and Mueller's ratchet hypotheses to explain the existence of sexual reproduction within the framework of social selection. In a population with two species which fit the same ecological niche, live in the same local environment, have the same degree of genetic diversity, but have different modes of reproduction, sexual and asexual, the sexual species will eventually dominate the local environment. This is due to asexual populations losing diversity for short-term adaptations to the environment.
Roughgarden proposes a population of dandelions which fit the above description. The parental generation of a sexually reproducing species and asexually reproducing species contains equal ratios of the three genotypes (A1A1, A1A2, and A2A2). The F1 generation of the asexual dandelions will contain the same ratio as the P-generation. Conversely, following standard principles of sexual reproduction, the F1 generation will be 25% A1A1, 25% A2A2, and 50% A1A2. With the addition of differential survival related to these genotypes (certain genotypes surviving better in different degrees of sunlight), the asexual population will eventually drift toward one genotype and die off when the environment changes to suit a different genotype. The sexual population in the same situation will remain diverse enough to survive changing environments.
From this theory, Roughgarden concludes that the main benefit of sexual reproduction is the maintenance of genetic diversity when compared to similar asexual populations.
The IR model for the development of anisogamy is named after its developers Priya Iyer and Joan Roughgarden. By considering the evolution of anisogamy in hermaphroditic marine invertebrates and bisexual plants, the theory postulates of a gene locus which controls both sperm and egg size produced by an organism. Anisogamy could evolve in diploid hermaphroditic adults as an individual adaptation which increases its own fitness.
Hermaphroditic animals and dioecious plants represent a large portion of sexually reproductive species. Under social selection theory, species where individuals produce two different gametes predate strictly gonochoristic and monoecious species. Separate sexes can, therefore, be described as derivations of primal hermaphrodites.
Males arising in primarily hermaphroditic species gain an advantage in certain environments as fertilizers because they lack the energy cost of producing eggs. The development of monoecious and gonochoristic species represents a transition from broadcast fertilization to localized and internal fertilization.
Simultaneous hermaphrodism exists in species with pre-Cambrian roots, and several families of organisms have shifted between hermaprodism and gonochoism over their evolutionary history. There are sequentially hermaphroditic species, such as the goby, which show bidirectional sex changing. The dwarf males of anglerfish in the family Ceratiidae function as "mobile testes" for the females of their species.
Animals help another in order to access reproductive opportunities. Any inequality in this opportunity is due to predation or resource availability. Therefore, there is value in boosting the reproductive fitness of an animals parents or siblings, both of which share genetic information. Even without this genetic relationship, reproductive transactions can be valuable. Sexual conflict arises from a failure for pairs to negotiate value of reproductive transactions effectively.
Mating can therefore serve purposes beyond reproduction, if the maintenance of social structures does not decrease effective fitness. Homosexual mating behavior is observed in species where this is the case. Several asexual species of whiptail lizards have been observed to engage in mating and pair-bonding despite the lack of gametic fusion.
Animal behavior can be understood as the intersection of three primary elements: genetic foundations, social systems, and individual reaction. A social state matrix is composed of genetic foundations and social systems to determine behavior arising from the intersection of them. For example, animals have genetics which determine reaction to potential foraging stimuli, but only search for the stimuli at certain times of day due to social systems. Therefore, social systems would be selected for which optimize behaviors such as foraging and mate selection.
Presented as an alternative to sexual selection theory, social selection has received criticism by defenders of sexual selection theory. Arguments have been made that Roughgarden anthropomorphizes animal behavior in order to suit her theory. Other critics argue that the holes in sexual selection theory which Roughgarden proposes, such as inconsistencies in male and female relationships and critiques of Bateman's principle, can actually be consolidated within the sexual selection framework.
The term "social selection" has been used by other researchers to describe elements of the selection process overlooked by the theory of sexual selection, and to view sexual selection as a subcategory of social selection. Mary Jane West-Eberhard used the term social selection to describe differential success in social competition for resources other than mates. Included in her theory are female competition for territory and competition for parental attention among offspring. Psychologist Geoffrey Miller has argued that if humans prefer altruistic sexual partners that would select for altruism directly, and evolutionary medicine researcher Randolph M. Nesse has argued that humans with altruistic tendencies are preferred as social partners from which derive fitness advantages, enabling humans as species of becoming extraordinarily cooperative and capable of creating culture.