The gardens were remade by André Le Nôtre from 1669 to 1673, and include a 2.4 kilometre long stone terrace which provides a view over the valley of the Seine and, in the distance, Paris.

The first castle, named the Grand Châtelet, was built on the site by Louis VI ca 1124. The castle was expanded by Louis IX of France in the 1230s. Louis IX's chapelle Saint Louis (Sainte-Chapelle) at the castle is a major work of the Rayonnant phase of French Gothic architecture. It served as a prototype for the Sainte Chapelle in Paris. In 1539 Francis I decided to rebuild Saint-Germain, using Pierre Chambiges as architect. This building is known as the Vieux Château; it was restored from 1855 by Eugène-Louis Millet, who, among other works, removed Hardouin Mansart's angle pavilions. Another section of the palace, the Château Neuf, was destroyed in 1777. Since 1867 the Vieux Château has housed the Musée des Antiquités Nationales de France (now Musée d'Archéologie Nationale).