The earliest origins of the imperial palace are probably in a royal hunting lodge, as Adam of Bremen mentioned for the Ottonian period. In 1005 Henry II erected a first imperial mansion in Goslar, which, due to the rich ore deposits under the nearby Rammelsberg, soon outstripped the nearby palace of Werla. In the 1030s Conrad II began to expand the site by laying the foundation stone for the Church of Our Lady. The district was completed and enjoyed its heyday under his son, Henry III. In 1048 Henry summoned to Goslar one of the foremost architects of his day, a man who later became Bishop of Osnabrück, Benno II. Under Benno's expert guidance the buildings that had been worked on since the 1040s were completed in the first half of the 1050s: a new Kaiserhaus, the one that we know today, and the Collegiate Church of St. Simon and St. Jude. Uncertain, however, is the date of the chapel dedicated to Saint Ulrich. It is believed to have been built either during the time of Henry III, Henry V or even Lothar III (from Süpplingenburg).