i have been told there was once a garden and a meadow between the house and the shore , where now lie only rubble and straggling seaweed , nobody supposing at the time of building that so much ground should ever be lost .