"They send us their Jews from Odessa, and we send them our Jews from Odessa."
Stern toured the Soviet Union in 1951, the first American violinist to do so. In 1967, Stern stated his refusal to return to the USSR until the Soviet regime allowed artists to enter and leave the country freely. His only visit to Germany was in 1999, for a series of master classes, but he never performed publicly in Germany.
Stern was married three times. His first marriage, in 1948 to ballerina Nora Kaye, ended in divorce after 18 months, but the two of them remained friends. On August 17, 1951, he married Vera Lindenblit (1927-2015). They had three children together, including conductors Michael and David Stern. Their marriage ended in divorce in 1994 after 43 years. In 1996, Stern married his third wife, Linda Reynolds. His third wife, his three children, and his five grandchildren survived him.
Stern died September 22, 2001 of heart failure in a Manhattan, New York, hospital after an extended stay.
In 1940, Stern began performing with Russian-born pianist Alexander Zakin, collaborating until 1977. Within musical circles, Stern became renowned both for his recordings and for championing certain younger players. Among his discoveries were cellists Yo-Yo Ma and Jian Wang, and violinists Itzhak Perlman and Pinchas Zukerman.
In the 1960s, he played a major role in saving New York City's Carnegie Hall from demolition, by organising the Citizens' Committee to Save Carnegie Hall. Following the purchase of Carnegie Hall by New York City, the Carnegie Hall Corporation was formed, and Stern was chosen as its first president, a title he held until his death. Carnegie Hall later named its main auditorium in his honor.
Stern served as musical advisor for the 1946 film, Humoresque, about a rising violin star and his patron, played respectively by John Garfield and Joan Crawford. He was also the featured violin soloist on the soundtrack for the 1971 film of Fiddler on the Roof. In 1999, he appeared in the film Music of the Heart, along with Itzhak Perlman and several other famed violinists, with a youth orchestra led by Meryl Streep (the film was based on the true story of a gifted violin teacher in Harlem who eventually took her musicians to play a concert in Carnegie Hall).
Stern maintained close ties with Israel. Stern began performing in the country in 1949. In 1973, he performed for wounded Israeli soldiers during the Yom Kippur War. During the 1991 Gulf War and Iraq's Scud missile attacks on Israel, he played in the Jerusalem Theater. During his performance, an air raid siren sounded, causing the audience to panic. Stern then stepped onto the stage and began playing a movement of Bach. The audience then calmed down, donned gas masks, and sat throughout the rest of his performance. Stern was a supporter of several educational projects in Israel, among them the America-Israel Foundation and the Jerusalem Music Center.
Among other instruments, Stern played the "Kruse-Vormbaum" Stradivarius (1728), the "ex-Stern" Bergonzi (1733), the "Panette" Guarneri del Gesù (1737), a Michele Angelo Bergonzi (1739–1757), the "Arma Senkrah" Guadagnini (1750), a Giovanni Guadagnini (1754), a J. B. Vuillaume copy of the "Panette" Guarneri del Gesu of 1737 (c.1850), and the "ex-Nicolas I" J.B. Vuillaume (1840). He also owned two contemporary instruments by Samuel Zygmuntowicz and modern Italian Jago Peternella Violins.
In 2001, Stern's collection of instruments, bows and musical ephemera was sold through Tarisio Auctions. The May 2003 auction set a number of world records and was at the time the second highest grossing violin auction of all time, with total sales of over $3.3M.