 The site only allowed for the building to be one room deep , as a workhouse and a barracks lay immediately behind . To exacerbate matters , there was a public right of way through the site to these buildings , which accounts for the access porticoes on the eastern and western sides of the façade . These had to incorporate columns from the demolished Carlton House and their relative shortness result in an elevation that was deemed excessively low , and a far cry from the commanding focal point that was desired for the northern end of the Square . Also recycled are the sculptures on the façade , originally intended for Nash 's Marble Arch but abandoned due to his financial problems . The eastern half of the building housed the Royal Academy until 1868 , which further diminished the space afforded to the Gallery . 
