The west front consists of the original castellated tower, four storeys in height, its windows enlarged to suit 17th-century taste; a second tower very similar to the first; and the two towers joined by a low screen of two storeys with the entrance gate in the center.

The palace is adjacent to the 12th-century abbey. The oldest part of the present building is the massive northwest tower, built in 1528-1532 by William Aytoun (active 1598; died ?1643) for James V. Except for this tower, the entire 16th-century palace was destroyed by an English army in 1544. The decision to re-create a palace on the site was taken by Charles II of England and Scotland (reigned 1660-1685). The contract to rebuild is dated March 1672.