The castle is situated on the eastern city of towering Castle Mountain at an altitude of 621.5 meters above sea level. The nearly triangular system measures the extent of 60 meters north-south, east-west 48 meters. Of the former castle, which is called the first time in 1129, there is no demand for construction judging. The walls still visible in places, technology refers to the late Romantic era.

Men were the first castle of Count Velburg. After they died in 1217, the castle fell to the Duchy of Bavaria. In the following years there were frequent seizures and changes in ownership, management of the ducal castle was made by judges or caregivers. By 1450 the castle burned down, but was completely restored.

Saying by the Cologne 1505, which ended the Bavarian Succession War, the castle came with the now at the foot of Castle Hill town originated at the newly established principality of Pfalz-Neuburg. 1507 Saint George Wispeck for distinguished service in the war of succession with the castle and lordship Velburg was granted tenure. After the extinction of the race in 1574 Wispeck relapse of the castle and town of Neuburg. Soon after, the nurses in the nursing home was built Wispeck Hans Adam settled in the city over the castle remained uninhabited.

At the capture of the castle in the Thirty Years War by General Haßwerth 1633, the castle was damaged but not repaired. The rapid decay directed the unauthorized removal of the roofs by the nurses Valentin Praun (1644-1650) one. Also, the chapel, dedicated to St. Pancras decayed.

1717-1721 were the stones of the castle used to expand the church and in the subsequent period for the construction of private homes, once this has been given to "landesherrlichem Consens' permission. 1793 the city bought the castle and Velburg the area of barren mountain castle in size from 17 hectares to 450 guilders from the National Electoral Directorate Neuburg.

During the 20th Century there were many, unfortunately, unsuccessful, approaches to conservation and preservation of remaining parts of ruin. Only in the years 1980-1986 was performed in an exemplary joint action to secure the ruin in its current form. It was almost ruined donjon made ​​on 12 meters again and now serves as a much-visited vantage point. Next wall and partially restores backups made ​​in the years 2003 to 2009.