data structures to recognize particular elements / shapes with similar patterns on the image in python
If you're planning on using something like PyTorch later on (which would tie in to the sort of neural-network functionality you're pursuing), I'd become familiar with how NumPy operates, as it bridges pretty well into Torch data structures. If it helps, a lot of SciPy and Matplotlib functions work beautifully with Numpy structures right out of the box.
It's hard to tell exactly what you're looking for in these non-neural data structures; where are you worried about performance concerns and bottlenecks?
I'd recommend starting out with some PyTorch (or other deep learning framework) tutorials regarding image recognition and classification; it will get you closer to where you want to end up, and you'll be better able to make decisions about what your eventual program structure needs will be.
I am having a trouble picking the right data structure as/library . I lack in experience in the area of image processing / pattern recognition . The aim is to building a simple prototype to learn recognizing particular shapes from construction plans. I would be great full for any indication about the data structure as I know It will be hard to switch it later on during the project and thus I am not entirely sure which one to pick. 
The problem is , I plan to use a kind of neural network / algorithm later on so the performance of processing of the data structure may happen to be my bottle neck.
I was thinking about NumPy / SciPy /  PIL / MatPlotLib
I will be extremely grateful for expertise of anyone who has tackled similar problem