The first castle built by the Sword Order named Wittenstein Castle ("made 
of white stone") was mentioned for the first time in 1209. It was located 
to the north from the bishop's castle. the castle was destoyed by townfolk 
in the war with the Order. Townpeople had to build a new castle for the 
Livonian Order at the site of the Holy Spirit hospital under condition to 
dismantle part of the town's fortification wall and to open the town from 
the castle's side. The tower of the Holy Spirit and a fragment of the 
fortification wall were added to the castle that was built under the 
supervision of master D.Kriege in 1330-1353. In 1484 townpeople destroyed 
this bilding as well but they were obliged to rebuild it and the construction 
was completed by 1515.
In the end of the XVI and in the XVII the castle was joined into the town's
fortification system. In the XVIII the mosat surrounding the castle was 
filled up and the square near the castle was constructed.
 
In the XVII-XVIII several wings were added to the castle as it was used as a
public building by polish and Swedish officials. In the XVIII it became a 
residence for Russian governor general. today a part of the building is 
going to be rebuilt for presenta tions.