The Castle is first mentioned in 1272, but probably dates back to the Lords of Prangins. It was rebuilt by Louis I of Savoy. The rectangular building was built in a variety of building styles. In 1463, it was extensively rebuilt. Around the same time, the square César Tower or Tour de Rive (now a residence), was built to defend the city. Following the Swiss Confederation invasion of Vaud in 1530, the Bernese bailiff was at Nyon. In 1574-1580 the castle was converted into the seat of the bailiff. It remained the seat of the bailiff and was renovated several times in the following centuries.