The abbey was founded in 1147 as a daughter house of Clairvaux by Robert, Earl of Gloucester, and was dedicated to the Blessed Virgin Mary. Early Christian crosses found in the close vicinity and conserved in the nearby Margam Stones Museum suggest the existence of an earlier Celtic monastic community. The founding abbot was William of Clairvaux. The third abbot, Conan, enjoyed the praise of Giraldus Cambrensis, whom he appears to have entertained prior to his official visit with Baldwin of Forde, Archbishop of Canterbury, to preach the Crusade in 1188. Conan (or Cunan) contributed to Patristic literature, as he is credited with the capitula or chapter-headings prefixing each section of St. Bernard's Sermons on the Song of Songs, one of the works for which that author was titled a Doctor of the Church.
The abbey was dissolved by King Henry VIII of England in 1536 and sold to Sir Rice Mansel. Significant holdings of the monastery library appear to have survived this event, including the Annales de Margan. At this time, only 12 monks were living in the monastery. From the Mansel family the abbey eventually passed into their descendants in the female line, the Talbot family. In the 19th century, C R M Talbot constructed a mansion at Margam Castle which overlooks the abbey ruins. The nave of the abbey continued in use as the parish church, as it does to this day. It is Anglo-Catholic in its churchmanship.
Margam Abbey now consists of the intact nave and impressive surrounding ruins. Those ruins not belonging to the church are now owned by the County Council. These remains, including the unusually large twelve-sided chapter house, dating from the 13th century, stand within 840 acre (3.4 km²) Margam Country Park, close to Margam Castle. The Abbey church of St Mary, the ruined Chapter House and the Abbey undercroft are all Grade I listed buildings.
On a hill overlooking the abbey stand the ruins of an outlying monastery building, Capel Mair ar y Bryn ("the chapel of St Mary on the hill"). The purpose of this building is thought to have been to allow members of the monastic community who were engaged in the keeping of flocks to fulfil their devotional obligations without having to return to the main church.