after the dutch left , things pretty much went downhill for a long time , and it got rougher and poorer after the civil war , when many veterans returned to the city , and an influx of immigrants vastly over-taxed the neighborhood housing .