Installing a swing set between two brick walls?
3 for two reasons:

Your walls are not meant for dynamic, changing loads such as those caused by swingset motions, so you don't know the potential damage long term; and

I would hesitate very much to buy a house which had a physical connection to another person's house via a toy.