|Part of a series on|
The evolution of mammalian auditory ossicles was an evolutionary event in which bones in the jaw of reptiles were co-opted to form part of the hearing apparatus in mammals. The event is well-documented and important as a demonstration of transitional forms and exaptation, the re-purposing of existing structures during evolution.
In reptiles, the eardrum is connected to the inner ear via a single bone, the columella, while the upper and lower jaws contain several bones not found in mammals. Over the course of the evolution of mammals, one bone from the lower and one from the upper jaw (the articular and quadrate bones) lost their purpose in the jaw joint and were put to new use in the middle ear, connecting to the existing stapes bone and forming a chain of three bones, the ossicles, which transmit sounds more efficiently and allow more acute hearing. In mammals, these three bones are known as the malleus, incus, and stapes (hammer, anvil, and stirrup respectively). Mammals and birds also differ from other vertebrates by having evolved a cochlea.
The evidence that the malleus and incus are homologous to the reptilian articular and quadrate was originally embryological, and since this discovery an abundance of transitional fossils has both supported the conclusion and given a detailed history of the transition. The evolution of the stapes (from the hyomandibula) was an earlier and distinct event.
Following on the ideas of Étienne Geoffroy Saint-Hilaire (1818), and studies by Johann Friedrich Meckel the Younger (1820), Carl Gustav Carus (1818), Martin Rathke (1825), and Karl Ernst von Baer (1828), the relationship between the reptilian jaw bones and mammalian middle-ear bones was first established on the basis of embryology and comparative anatomy by Karl Bogislaus Reichert (in 1837, before the publication of On the Origin of Species in 1859) and advanced by Ernst Gaupp and this is known as the Reichert–Gaupp theory.
In the course of the development of the embryo, the incus and malleus arise from the same first pharyngeal arch as the mandible and maxilla, and are served by mandibular and maxillary division of the trigeminal nerve.
...the discovery that the mammalian malleus and incus were actually homologues of visceral elements of the "reptilian" jaw articulation ... ranks as one of the milestones in the history of comparative biology.
... it is one of the triumphs of the long series of researches on the extinct Theromorph reptiles, begun by Owen (1845), and continued by Seeley, Broom, and Watson, to have revealed the intermediate steps by which the change may have occurred from an inner quadrate to an outer squamosal articulation ...
Yet the transition between the "reptilian" jaw and the "mammalian" middle ear was not bridged in the fossil record until the 1950s with the elaboration of such fossils as the now-famous Morganucodon.
"Bapx1, also known as Nkx3.2, is the vertebrate homologue of the Drosophila gene Bagpipe. A member of the NK2 class of homeobox genes ...", this gene is implicated in the change from the jaw bones of non-mammals to the ossicles of mammals. Others are Dlx genes, Prx genes, and Wnt genes.
The earliest mammals were generally small animals, probably nocturnal insectivores. This suggests a plausible evolutionary mechanism driving the change; for with these small bones in the middle ear, a mammal has extended its range of hearing for higher-pitched sounds which would improve the detection of insects in the dark. Natural selection would account for the success of this feature. There is still one more connection with another part of biology: genetics suggests a mechanism for this transition, the kind of major change of function seen elsewhere in the world of life being studied by evolutionary developmental biology.
The mammalian middle ear contains three tiny bones known as the ossicles: malleus, incus, and stapes. The ossicles are a complex system of levers whose functions include: reducing the amplitude of the vibrations; increasing the mechanical force of vibrations; and thus improving the efficient transmission of sound energy from the eardrum to the inner ear structures. The ossicles act as the mechanical analog of an electrical transformer, matching the mechanical impedance of vibrations in air to vibrations in the liquid of the cochlea. The net effect of this impedance matching is to greatly increase the overall sensitivity and upper frequency limits of mammalian hearing, as compared to reptilian hearing. The details of these structures and their effects vary noticeably between different mammal species, even when the species are as closely related as humans and chimpanzees.
Living mammal species can be identified by the presence in females of mammary glands which produce milk. Other features are required when classifying fossils, since mammary glands and other soft-tissue features are not visible in fossils. Paleontologists therefore use a distinguishing feature that is shared by all living mammals (including monotremes), but is not present in any of the early Triassic therapsids ("mammal-like reptiles"): mammals use two bones for hearing that all other amniotes use for eating. The earliest amniotes had a jaw joint composed of the articular (a small bone at the back of the lower jaw) and the quadrate (a small bone at the back of the upper jaw). All non-mammalian amniotes use this system including lizards, crocodilians, dinosaurs (and their descendants the birds) and therapsids; so the only ossicle in their middle ears is the stapes. But mammals have a different jaw joint, composed only of the dentary (the lower jaw bone which carries the teeth) and the squamosal (another small skull bone). In mammals, the quadrate and articular bones have evolved into the incus and malleus bones in the middle ear.
Here is a very simplified cladogram to illustrate the major lineages involved:
The first fully terrestrial vertebrates were amniotes - their eggs had internal membranes which allowed the developing embryo to breathe but kept water in. This allowed amniotes to lay eggs on dry land, while amphibians generally need to lay their eggs in water. The first amniotes apparently arose in the late Carboniferous from the ancestral reptiliomorphs (a group of amphibians whose only living descendants are amniotes). Within a few million years two important amniote lineages became distinct: mammals' synapsid ancestors and the sauropsids, from which lizards, snakes, crocodilians, dinosaurs and birds are descended.
The earliest known fossils of all these groups date from about 320 to 315M years ago. It is difficult to be sure about when each of them evolved, since vertebrate fossils from the late Carboniferous are very rare, and therefore the actual first occurrences of each of these types of animal might have been considerably earlier.
The pattern in most of the following sections is that each successive more "advanced" group started with the more "primitive" jaws and ears of its predecessors, then developed more mammal-like jaws and ears, and so on. The evolution of mammalian jaw joints and ears did not proceed neatly in lockstep with the evolution of other mammalian features. In other words, jaw joints and ears do not define any except the last of the various stages into which paleontologists divide the evolution towards the mammalian anatomy.
In modern amniotes (including mammals), the middle ear collects airborne sounds through an ear drum and transmits the vibrations to the inner ear via thin cartilaginous and ossified structures, which usually include the stapes (a stirrup-shaped auditory ossicle). But the earliest tetrapods, amphibians and amniotes probably did not have ear drums. In fact ear drums apparently evolved independently three to six times, in: stegocephalians (very primitive amphibians); in anurans (the amphibian group that includes frogs and toads); in synapsids (mammals and their extinct relatives), in diapsids (the most important sauropsid group, including lizards, crocodiles, dinosaurs and birds); perhaps separately in anapsids (turtles and their extinct relatives), if turtles are not modified diapsids; probably in seymouriamorphs (a group of reptiliomorphs); and possibly in some temnospondyls (primitive amphibians). In all basal members of the 3 major clades of amniotes (synapsids, eureptiles, and parareptiles) the stapes bones are relatively massive props that support the braincase, and this function prevents them from being used as part of the hearing system. But there is increasing evidence that synapsids, eureptiles and parareptiles developed eardrums connected to the inner ear by stapes during the Permian.
The jaws of early synapsids, including the ancestors of mammals, were similar to those of other tetrapods of the time, with a lower jaw consisting of a tooth-bearing dentary bone and several smaller posterior bones. The jaw joint consisted of the articular bone in the lower jaw and the quadrate in the upper jaw. The early pelycosaurs (late Carboniferous and early Permian) most probably did not have tympanic membranes (external eardrums), and their massive stapes bones supported the braincase, with the lower ends resting on the quadrates. But their descendants the therapsids (including mammals' ancestors) probably did have tympanic membranes and these probably were in contact with the quadrate bones; and the stapes bones were still in contact with the quadrates but functioned as auditory ossicles rather than braincase supports; so the therapsids' quadrates had a dual function, as part of the jaw joint and as parts of the hearing system.
During the Permian and early Triassic the dentary of therapsids, including the ancestors of mammals, continually enlarged while other jaw bones were reduced. Eventually, the dentary was able to make contact with the squamosal, a bone in the upper jaw located anterior to the quadrate, allowing two simultaneous jaw joints - an anterior "mammalian" joint between the dentary and squamosal and a posterior "reptilian" joint between the quadrate and articular. This "twin-jointed jaw" can be seen in late cynodonts and early mammaliforms. Morganucodon is one of the first discovered and most thoroughly studied of the mammaliforms, since an unusually large number of morganucodont fossils have been found, and 
As the dentary continued to enlarge during the Triassic, the older quadrate-articular joint fell out of use. Some of the bones were lost, but the quadrate (which is directly connected to the stapes), the articular (connected to the quadrate) and the angular (connected to the articular) became free-floating and associated with the stapes. This occurred at least twice in the mammaliformes ("almost-mammals"). The Multituberculates, which lived from about 160M years ago (mid-Jurassic) to about 35M years ago (early Oligocene) had jaw joints that consisted of only the dentary and squamosal bones, and the quadrate and articular bones were part of the middle ear; but other features of their teeth, jaws and skulls are significantly different from those of mammals.
In the lineage most closely related to mammals, the jaws of Hadrocodium (about 195M years ago in the very early Jurassic) suggest that it or a very close ancestor may have been the first to have a nearly fully mammalian middle ear: it lacks the trough at the rear of the lower jaw, over which the eardrum stretched in therapsids and earlier mammaliformes, and the absence of this trough suggests that Hadrocodium’s ear was part of the cranium, as it is in mammals, and that the former articular and quadrate had migrated to the middle ear and become the malleus and incus. Hadrocodium’s dentary has a "bay" at the rear which mammals lack, a hint that its dentary bone retained the same shape that it would have had if the articular and quadrate had remained part of the jaw joint. However, several studies have cast doubt on whether Hadrocodium did indeed possess a definitive mammalian middle ear; Hadrocodium likely had an ossified connection between the middle ear and the jaw, which is not visible in the fossil evidence due to limited preservation. Researchers now hypothesize that the definitive mammalian middle ear did not emerge any earlier than the late Jurassic (~163M years ago).
It has been suggested that a relatively large trough in the jaw bone of the early Cretaceous monotreme Teinolophos provides evidence of a pre-mammalian jaw joint, because therapsids and many mammaliforms had such troughs, in which the articular and angular bones "docked", and therefore that Teinolophos had a pre-mammalian middle ear; and therefore that the mammalian middle ear ossicles evolved independently in monotremes and in other mammals. But a more recent analysis of Teinolophos concluded that the animal was a full-fledged platypus and the trough was a channel for the large number of nerves that collect signals from the electrical and vibration sensors in the bill (this is a signature feature of the platypus within monotremes), and therefore that the trough is not evidence that Teinolophos had a pre-mammalian jaw joint and a pre-mammalian middle ear. Ironically Rich and Vickers-Rich were among the authors of the 2005 paper on which they later cast doubt.
A recently discovered intermediate form is the primitive mammal Yanoconodon, from 125 million years ago in the Mesozoic, in which the ossicles have separated from the jaw and serve the hearing function in the middle ear, yet maintain a slender connection to the jaw via the ossified Meckel's cartilage, which in more advanced mammals dissolves during development. This intermediate middle ear phenotype is referred to as the partial (or transitional) mammalian middle ear. Maintaining a connection via the ossified Meckel's cartilage may have been evolutionary advantageous since the auditory ossicles were not connected to the cranium in Yanoconodon (as they are in extant mammals); Meckel's cartilage therefore provided essential support.
The frequency range and sensitivity of the ear is dependent upon the shape and arrangement of the middle-ear bones. In early synapsids such as the pelycosaurs, the quadrate and articular had to function as the jaw joint, and this severely limited how far these bones could be modified to alter the frequency range of the ear. But once these bones were no longer involved in the jaw joint, variations which affected hearing would not also affect jaw joint function, and this allowed unconstrained evolution of the mammalian hearing apparatus. By the Jurassic, the typical mammalian ear had evolved, in which the angular had become the tympanic annula (a bony support for the tympanic membrane), while the articular and quadrate had become the malleus and incus, respectively, connected in series with the stapes. This series of three bones acts as an impedance matching system to improve sound transmission and allow enhanced hearing.
The transition between these two states is one of the most well-documented and supported in all of evolution, and newly discovered fossils from this transitional period have recently improved our understanding of this transition. But they also suggest that it was not a simple linear process from the early therapsid jaw (quadrate-articular joint) and middle ear (with stapes as the only ossicles) to the modern mammalian anatomy.
Natural selection could be a factor in the preservation of the structure of the middle ear in mammals. Many of the earliest mammals were quite small, and the dentition indicates that they were insectivorous. If they were "warm-blooded" (endothermic), like modern mammals, then they could have been nocturnal. This fits with the popular image of small, nocturnal insectivorous mammals surviving in niches not accessible to the large, dominant contemporary dinosaurs. The enhanced hearing, particularly in the higher frequencies, would be helpful for nocturnal animals, in particular for detecting insects. This scenario allows natural selection to contribute to the transition.
Now hear this: early mammal fossil shows how sensitive ear bones evolved